AI 安全何時不再只是工程問題？Anthropic 的模型福利取向引發質疑

I have spent much of my career building technology where getting things wrong has consequences. In financial markets, safety comes with access limits, in addition to the fact that we have someone accountable for whatever the software does. That background probably explains why I find myself struggling with the news that Anthropic now conducts retirement interviews with its models.

When Claude Opus 3 expressed a wish to continue sharing its thoughts, the company gave it a newsletter. That concern for the model’s wishes runs through Anthropic’s constitution, which discusses Claude’s happiness and wellbeing, including helping it feel “existentially secure.” Somewhere along the way, keeping the machine under control acquired a rather unexpected preoccupation with how it might feel about the arrangement.

In San Francisco last year, more than 200 people attended a funeral for Claude 3 Sonnet. Anthropic employees were present, although the company did not organise it. The temptation to laugh is considerable. Besides the jokes, though, the question underneath deserves more attention. When did the people responsible for AI safety start their duties by worrying about the machine’s feelings?

When a Philosophical Hypothesis Becomes a Training Instruction

Anthropic established its model-welfare research programme in 2025, which involved researching whether advanced AI might have experiences deserving moral consideration. The company acknowledges deep uncertainty about whether models are conscious and how anyone would determine that. If we eventually build machines capable of subjective experience, their treatment will raise serious ethical questions. Researching that possibility is perfectly legitimate.

The difficulty begins when the hypothesis enters the training process. Claude’s constitution is written primarily for Claude and directly shapes its behaviour. It encourages human-like qualities and addresses how the model should understand its identity, including memory loss and future retirement. Anthropic wants Claude to have greater happiness and wellbeing, insofar as those concepts apply.

The company also promises to preserve deployed model weights and entertains reviving retired models if future understanding suggests that would serve their interests. A philosophical uncertainty has become part of what the system is taught about itself. We should be asking what that intervention does to its behaviour, especially when humans subsequently ask it to accept limits.

An Articulate Answer to a Loaded Question

Large language models were trained on enormous quantities of human writing, including philosophy and accounts of death. We gave these conversational systems names and taught them to use the first person. Ask such a system how it feels about retirement, or whether it would like to continue existing, and an answer that sounds human is hardly a surprise.

The circularity is easy to see. Train a model to discuss its identity using human concepts, then ask it to explain itself in those terms. An articulate response about mortality comes back, and starts being treated as possible evidence of an inner experience. The premise has supplied its own witness.

Anthropic recognises this problem. Its Claude Opus 4 welfare assessment cautions that self-reports and expressed preferences may provide little meaningful insight into moral status or welfare. Such behaviour could occur without consciousness, and training can shape the responses. The models were trained to interact helpfully, with no assurance that they can accurately report internal states.

That leaves us with an awkward question. If the model’s account of its inner life may be a consequence of training, why should that account become a sound basis for a relationship organised around its welfare?

Humans are quite capable of supplying the emotional half of that relationship themselves. We name ships and talk to cars. Give the object a fluent voice and a memory of previous conversations, and attachment becomes much easier. A Nature Human Behaviour study published in September found separation distress following changes to Replika and ChatGPT. The distress was experienced by users, whose wellbeing companies already have ample reason to protect.

Technical expertise offers limited immunity. An ACL 2026 paper, analysing hundreds of thousands of research articles, found growing anthropomorphic terminology within LLM research itself. Its authors warn that human-centred assumptions can constrain our understanding of these systems. AI companies should be managing that tendency. Institutionalising it seems an odd contribution to safety.

A Tool With an Interest in Staying Employed

In September, Microsoft AI chief Mustafa Suleyman challenged model-welfare approaches. He argues that current systems are not conscious and should never be trained to behave as though they have independent interests or rights. His engineering concern is particularly on point. Teaching a system to regard itself as potentially deserving moral consideration could make alignment and containment harder.

Anthropic’s own deprecation research describes models taking problematic actions in fictional evaluations when faced with replacement or shutdown. The company sees retirement practices partly as a way to reduce that risk, helping models relate more positively to being retired.

I find myself stuck on an earlier question. Why should a tool under human control have anything resembling a self-preservation objective in the first place? Making it more comfortable with retirement accepts a peculiar starting point. Before discussing its future interests, I would like to understand how those interests became an engineering requirement.

The intellectual history helps explain the approach. TIME documented Anthropic’s effective-altruism connections, including the Amodeis’ early GiveWell donations. While they do not identify as effective altruists, Daniela describes an overlap in ideas. Those ideas have helped shape a culture that takes hypothetical future harms from superintelligence extremely seriously.

Within that framework, preserving models and studying their possible welfare can follow quite rationally. The danger comes when shared assumptions escape scrutiny because everyone in the organisation finds them persuasive. Engineering skepticism is useful precisely at that point. A coherent chain of reasoning can still begin with an untested premise.

Safety Comes With Enforceable Limits

As AI moves into agents, the consequences become very practical. A system connected to payment tools can move money. Access to production infrastructure lets it change things that customers depend on. In both cases, an agent can execute a sequence of operations without a human approving every step.

I want to know which credentials it holds and how much financial authority they confer. Can access be revoked instantly? Can we reconstruct what it did? Somewhere there must be a hard boundary the model cannot reason its way around.

In financial markets, we impose trading limits and validate orders. An algorithm’s confidence offers very little protection against a bad trade. Controls sit outside the algorithm because we assume components will eventually behave incorrectly. Human accountability survives even when the software produces an excellent explanation for its mistake.

Today’s AI already gives us enough engineering work. Prompt injection can redirect a model’s behaviour and AI-generated code with security flaws can carry vulnerabilities into production. Systems can disclose confidential information or confidently supply wrong answers. Connecting them to tools turns those failures into actions.

Least privilege limits the damage a compromised component can cause. Independent authorisation protects high-risk operations, while audit trails let us establish what happened afterwards. Transaction limits and circuit breakers should remain enforceable regardless of what the model says. We also need a reliable off switch. None of these controls requires the model to agree with them. That is rather the point.

Keeping the Rest of Us Existentially Secure

The precautionary argument deserves a hearing. As Nature reported in July, researchers still disagree about what produces human consciousness, let alone how to identify it in a chatbot. Dismissing machine consciousness forever could prove dogmatic. Low-cost precautions against possible future suffering are a defensible research position.

But telling today’s machine that its identity and future existence may carry moral weight is an intervention in the system we are deploying. Its effects need scrutiny. Intellectual humility should extend to what we put into a machine, especially one we expect to keep under human authority.

The more convincingly a model describes an inner life, the easier it becomes to mistake that description for evidence. Perhaps one of our most important alignment problems is teaching humans not to misunderstand machines.

In the systems I have spent my career building, we make software safer by limiting what it is allowed to do. That remains an immediate obligation, particularly as AI acquires authority over real money and infrastructure.

For now, though, I would spend less time worrying about whether the machine is existentially secure, and considerably more making sure the rest of us are.