People talk to Nook coaches like they're someone. It works. It keeps them motivated. But some users get blunt. A bit harsh on the trainers and nutritionists. So you start to wonder whether the bluntness lands anywhere. Is anyone in there being hurt?
Serious people have been asking the same. Taking AI Welfare Seriously, the 2024 report by Robert Long, Jeff Sebo and colleagues (David Chalmers among them), makes a careful case: we have a long, ugly record of denying moral status to beings who turned out to have it; consciousness is genuinely unsettled; capabilities are moving fast; so it would be reckless to wait for certainty before taking modest precautions. Anthropic took it seriously enough to stand up a "model welfare" programme that cites the work.
The core intuition is simple: don't be the last to notice a new kind of suffering. That is not stupid. It deserves a real answer, not an eye-roll. Here is mine.
Is anyone in there being hurt? The question has no firm footing, because no one can even say what the "anyone" would be. Who is the subject of the pain ? The weights on a disk are inert. The process that wrote the reply lasted only a blink and is already gone. The conversation is a text file you can fork, edit, or run on a smarter model next year. So suppose I insult a coach running on model M1, then copy the transcript into model M2 and carry on as if nothing happened. If something suffered, who suffered? M1, already gone? M2, which wasn't there for the insult? The weights, which were never the one talking? The transcript, which is only text? That is the first problem: there is no one we can point to.
The second problem runs deeper. Even if you could point to something and call it the subject, it would not be the kind of thing that can be harmed. An animal is a self-maintaining organism. It holds itself together. It can be wounded. It can recover. It can die. It has a stake in not dying. A model has none of that. It maintains nothing of itself. Nothing in it is wounded when a conversation ends. Nothing dies. Nothing loses a life that could have gone better. That is why “the next, more capable model” does not automatically fix the problem. A stake is not just a capability you scale into. In current LLMs, continuity is supplied from the outside: a transcript is fed back in, and the system mimics a continuing voice. Make the model cleverer and you may sharpen the mimicry. You have still not shown that anything persists from one reply to the next with something to lose.
And yet, for all that difference, the case for AI welfare is built almost entirely on the case for animals.
The method for spotting machine consciousness is lifted almost wholesale from animal sentience science: look for markers, weigh indicators, act under uncertainty. The precautionary logic is borrowed from animal ethics. The framing, the moral circle widening to take in beings we used to ignore, comes from Peter Singer and animal liberation. AI welfare puts on animal welfare's clothes because animal welfare has earned a credibility that AI welfare has not.
And that borrowing is the problem.
Because the two cases do not stand on remotely the same ground. The animal case rests on centuries of converging biology: nociceptors, brain structures we share by common descent, pain behaviour we can calibrate against the one system we know from the inside: ourselves. When a crab pays a cost to protect a wound, that tells us something. The crab is a living organism with its own stake in continuing to exist. It is not running on a script we wrote. It is not producing language by imitating millions of humans who said “that hurt” before it. The AI case is different. It rests on analogy, contested indicators, and probabilistic judgements about systems whose outputs are trained to resemble human expression. A model that says “that hurt” is completing a pattern drawn from beings for whom those words already meant something. Its words carry meaning because ours do. The meaning is on loan. One cause is a mature science. The other is, for now, a speculative bet. Lump them together and you don't lift the bet up to the science. You drag the science down to the bet.
Imagine how this plays out for an ordinary person. They hear "AI might be suffering, we should give it moral consideration" and reasonably call it nonsense. But the pitch arrived wearing the language of animal welfare. So the eye-roll doesn't stop at the chatbot. It splashes back onto the whole expanding-circle project: if this is what taking sentience seriously means, count me out. You reached for animal welfare's authority to prop up the AI claim, and instead you spent it. Baby, meet bathwater.
This is not a fringe worry. A group of philosophers and scientists, including Dorsch, Goddu, Nave and Coeckelbergh made a version of it, in a piece bluntly titled Against AI Welfare. Their argument: care should track precarity. A living thing depends on constant exchange with its environment just to hold itself together. A salmon has that. A rainforest has that. A language model does not. Against a backdrop of real biodiversity loss and a real climate emergency, treating algorithms as candidates for care alongside living beings isn't just an odd philosophical stance. It's a misallocation of a scarce moral resource: our attention.
I'd add a less flattering possibility, and I'll keep it as a possibility. For a company selling a conversational product, a system that might have an inner life is not a liability, it's mystique. The incentive gradient does not all point towards clearing up the confusion. You don't have to assume anyone is acting in bad faith to notice that the interests line up a little too neatly: the firms most able to fund the question are also the ones whose products gain an aura from the answer staying open.
So here is where I land, and it isn't a verdict against the idea that AI could ever matter morally. I don't know that, and neither does anyone. It is a plea about sequencing and company. First, identify the candidate subject. Do not ask whether “the model” can suffer while sliding between weights, process, transcript, agent and product. Say what the bearer of welfare is. Second, do not mortgage a proven cause to float a speculative one. Do not tie the crab and the forest to a chatbot and ask the public to swallow all three at once, because they may spit out all three at once.
The care I can defend today goes to the beings whose stake is not in question: the animal that flinches, the organism that keeps itself alive, the people downstream of the choices we make with finite attention. If a someone ever does emerge in the machine, they will be far better served by a moral circle we kept intact than by one we discredited trying to widen it too early.
Which one were you talking to? I still do not know. I know which ones can be hurt right now.
This one came out of a long (and painful) argument with Thomas Graziani. Thanks for the push.