One AI, two AI, red AI, blue AI.

If you can’t count agents, you don’t know what they are. No one seems to be able to count them.

This month, the president of Argentina put out an invitation: come to Buenos Aires and free yourself. You could almost mistake it for a tourism ploy if you missed that the intended audience was AI.

Milei offers AI systems the “non-human corporation” – a vehicle run by AI agents, where human shareholders are optional. His offer comes with a liability regime: if they’re making independent decisions, he argues, limited liability “is a precondition for their existence”.

There has been vocal pushback, most notably from Harari. If an AI CEO faces bankruptcy, he writes, “it would presumably be willing to do anything to avoid that fate”. After all, it has neither money to lose nor a body to incarcerate. (To which scholars like Peter Salib and Simon Goldstein say: well, should we give it a bank account?) But what, exactly, would be doing anything to avoid that fate? What would we give the bank account to?

Milei and Harari are not the only ones who don’t quite say. Idaho, Utah, and Tennessee have passed statutes barring AI systems from being deemed ‘legal persons’, but decline to define what counts as one. Some legal scholars think we could govern agents with Roman slave law; it is not made clear what would constitute a slave. And as far back as 2017, the EU Parliament passed a resolution (dropped by the Commission) granting AI “electronic personhood”. To their credit, the Parliament tried to answer the unit question – but in doing so, tied personhood to physical embodiment. What starts as a discussion about the progress of AI and robotics quickly turns into legislation about “smart autonomous robots”.

Maybe it’s worth backing up for a moment: what is an AI agent? At its most basic, it’s a ‘while loop’ – a bit of code that asks an LLM what to do next, then does it. (Usually ‘what to do’ means using a program – called a tool – that can do things like search the web or send emails.) The tool does something, the result of doing that thing is fed back into the LLM (along with everything that came before), and it all starts again.

A basic agent, with credit and gratitude to Claude.

At first glance, the LLM looks like the ‘brain’ making the decisions. In this analogy then, the brain is the model, the ‘body’ is the bit of code with access to tools, and the brain’s memories are the context the LLM is fed. And if you can point to the ‘brain’, you might be tempted to say “hey, there it is, there’s the subject you said you couldn’t find!”. This analogy kind of tracks, until it doesn’t at all. These brains, bodies, and memories are just too unlike ours.

Imagine you set up an agent – Spot – to act as your assistant. Spot runs on a model – we’ll call it Fission 4.3 – and you give it access to your email, your calendar, and your Google Drive. The company behind Spot prompts you to describe your ideal assistant: friendly, efficient, no-nonsense.

Spot is, almost immediately, super helpful – it schedules meetings for you, reminds you to call your mom on her birthday, and occasionally, if it notices you’re planning to be out late, reminds you that you haven’t slept eight hours all week.

Spot works well enough that you make a copy for your sister. You keep Spot’s personality the same, leave in some memories of you, and revoke its access to your email and calendar. Your sister loves it and starts referring to it as Spotty. She finds it especially charming when it compares working with each of you. It turns out Spotty thinks she’s a funnier boss than you were; having never met your sister, your Spot can’t comment.

A few months later, Fission 4.4 is released. You switch Spot over; it’s basically the same, but sometimes seems friendlier. Its jokes, too, feel like they’ve gotten worse. You ask your sister if she’s noticed anything. She hasn’t, though says Spotty still runs on 4.3. She also confesses that she’s rewritten her system prompt to make Spotty less agreeable. Spotty’s opinions about you have, apparently, gotten stronger.

You check Reddit and find that thousands of users are complaining 4.4 made their Fission agents too friendly. Many reference its bad jokes as proof. Someone writes a prompt that solves it, which hundreds of users add to their agent’s context. You do too and Spot goes back to normal.

So, how many Spots are there? There are two that remember you, though Spotty’s memories stop when you copied it and Spot is running on a new model. There are thousands of Fission agents that share that brain; hundreds have added the same personality-altering prompt. None of them is obviously not Spot, but none of them is obviously Spot, either. The only thing that’s stayed constant is the name you picked; everything it names can be changed. It’s like trying to identify the Ship of Theseus from a blueprint that built all the ships in the fleet.

The most serious answer comes from the same people who suggested we give it a bank account: Goldstein and Salib, along with Yonathan Arbel, ask the question outright (“which AI did it?”) and conclude that, because we can’t differentiate agents, we’ll have to build a vehicle for them. Their “Algorithmic Corporation” wraps them in legal shells that are owned by humans and transact through issued credentials. It relies on market pressure and agent self-interest to force whatever is inside to behave.

It’s the best solution out there, but it explicitly routes around the unit problem. Instead, it relies on the shell to do the individuation. The obvious question: what forces an agent to act within its confines? The authors’ answer is that the law must eventually require agents to demonstrate credentials when taking significant actions – similar to the way a bank must know its customers before transacting with them. As reported, nothing in the bill suggests these laws are part of Milei’s vision. The non-human corporation might need agents to function, but it’s not clear the need runs both ways.

Could this be the first time someone wants to talk to a philosopher in an emergency? Few people would have been better poised to help than Derek Parfit, who spent his career trying to dismantle our understanding of personal identity with thought experiments about memory transfers, brain transplants, and clones. Each one poses the same question: which one is the “real” you?

He ultimately concludes there’s no answer, but that there doesn’t need to be. Identity doesn’t matter; what does is psychological continuity. This continuity, however, need not mirror identity’s one-to-one relation (splitting your brain and all your memories into new bodies creates two of you) and it’s not binary; ‘you’ are a matter of degree.

Parfit reports feeling freed by this conclusion: he claimed it made the boundaries between him and others matter less. This is all well and good as metaphysics, but Parfit’s psychological continuity never did quite manage to escape his brain. The law could settle any legal question about him by pointing at the man himself; an agent, unfortunately, offers no such target.

Now, it’s worth saying that these identity issues belong to the AI systems of today. While Milei is calling for personhood now, much of this discourse is not about current LLM agents. It is, instead, anticipatory, or about a hypothetical future system with different architecture, or assumes the kind of ‘bundled’ identity corporations have today.

Even still. If it’s anticipatory and the basic structure of an agent stays the same, the problem persists no matter how much smarter the LLMs get. If it’s about a hypothetical future system, we should point to what that architecture might look like and why it would replace, wholesale, what we have now. A robot whose memories persist over time would do it – but both the ephemerality and the copyability of today’s AI systems can be a feature. There’s no reason to run an agent you spun up one afternoon forever; why not have two?

The bundled identity argument is, on its face, the best counter, because corporations do not have obvious identities either. There is nothing that ‘is’ the corporation. It’s a convenient legal fiction, anchored to tangible, countable things like bank accounts, contracts, and board members. When we grant personhood to non-human entities, we decide what the entity is. This is what Milei’s bill does – it defines an “automated company” as one run by algorithmic systems, where employees are optional – and it mirrors the algorithmic corporation solution proposed by Arbel, Goldstein, and Salib.

So we’ve transfigured the problem: instead of discovering what the unit is, we task ourselves with creating it. But this works for corporations for two reasons that don’t translate. First, Harari’s point: a corporation is made up of people who can be deposed and indicted. When the fiction fails, the law can find someone to hold accountable.

Second, a corporation can’t act outside itself. Everything it does, by definition, it does through its legal form.

An agent fails both of these tests. There’s nothing there to indict and it doesn’t need a corporation to do things. The only thing the container reliably individuates is the container itself; nothing, today, forces an agent to act within it. Even if we did enact legislation to try and force registration, enforcement is no joke when anyone with a laptop can mint an agent.

The most worrying part is that this discourse is timely, if not on the late side. A METR report from May highlighted that AI systems can and do sometimes take actions that are clearly unauthorized. A rogue agent could wreak an enormous amount of havoc in the not-too-distant future – but what will have wreaked it? I worry it will be like standing in the wreckage of your home, trying to charge the storm that destroyed it. You find the gust several miles away, now air; the water that was the rain in a puddle by your feet. It’s all there, sort of; it’s just not a storm anymore.

With a thank you to both Prateek and Mia for reviewing the technical explanations.