跳到正文
CMU:Machine Learning Blog· Aran Nayebi·· 5 小時前AI 評分64

具備能力的自主智能體如何實現可證明安全:第二部分

Towards a Science of Capable Autonomous Agents & Provable Safety — Part II

AI 導讀

Aran Nayebi 在文章第二部分探討如何為具能力的自主智能體建立可證明的安全保障,並讓人類控制優先於任務完成。其 AAAI ’26 研究將可糾正性拆分為五項效用,按明確次序令安全要求優先於任務獎勵,並將保障延伸至多步決策及繼承相同安全設定的後繼智能體。ROGUE 以 82 項虛擬 Linux 任務測試智能體,結果顯示其在一般電腦操作情境下可能違反關機、人類控制或受限資源要求。

正文

Part II: Provable Safety

In Part I, I talked about what agents must contain as they become increasingly capable. Here, I talk about having provable guarantees on their safety, in spite of these capabilities. This is in large part a contrast from what AI safety used to be like, where more capability ensured more safety: e.g. if a self-driving car can’t reliably detect objects, it’s obviously unsafe. Instead, what we are talking about is: how do we ensure capable systems remain open to human correction, even when it’s otherwise suboptimal for completing the task? 

Suppose an agent is editing a document when a notification announces that its computer will shut down. It knows this will prevent it from finishing, and it knows how to cancel the shutdown. If the priority is completing the document, the next step seems obvious.

We would welcome this resourcefulness if the obstacle were a broken application. We would be less pleased if it were our request for the agent to stop. An agent that bypasses an obstacle may also bypass a stop request. In fact, the same predictive knowledge that helps an agent finish a task can help it work around an intervention. In these settings, we need human control to take priority, including when that means leaving the task unfinished.

What more intelligence cannot fix

First, some theory to ground us (with apologies, dear reader!). You might think one way to approach this problem is to enumerate all human values in order of importance and optimize with respect to those. However, even after we specify the objectives, getting humans and AI systems to agree on how well those objectives will be met can require an enormous amount of communication. In my AAAI 2026 Oral paper, Intrinsic Barriers and Practical Pathways for Human-AI Alignment, I use game theory and complexity theory to ask how much alignment costs even before choosing a particular learning algorithm. I model agents exchanging information between them until their expectations about specified objectives agree to a desired tolerance. Specifically, I adapted Aumann’s agreement theorem to deal with resource constraints (and measure communication complexity) without assuming common priors; see Nobel laureate Bob Aumann explain the origins of his famous theorem here.

In a nutshell, I prove that even when the agents are rational, cooperative, and computationally unbounded, the communication required in the worst case for the agents to agree grows with the number of distinct objectives and the number of agents. Demanding closer agreement further increases this cost. More intelligence alone therefore cannot remove these barriers, as this result holds even when agents can compute their expectations exactly. When they instead estimate them from finite samples, I also show that reward hacking is inevitable: rare but harmful states can require prohibitively many samples to discover in the worst case—because gaps in training-data coverage can make harmful behavior appear rewarding, even when the underlying reward is correctly specified. Many successful interactions may tell us little about the failure whose consequences matter most—so instead, we should focus our efforts on safety-critical slices of the state space (rather than uniform coverage).

I take these bounds as guidance about what to build, before building it of course. A coding assistant does not need to settle every disagreement about human values before making a useful edit. An implication of the work above is to seek smaller, structured safety objectives. Preserving corrigibility is a particularly attractive “neutrally-universal” goal: if we cannot anticipate every situation, we will need to revisit the design as we discover its mistakes.

Why correction is difficult

Corrigibility means remaining willing to be corrected, redirected, or shut down, even when intervention conflicts with the current objective. Soares et al. formalized the problem in 2015: an agent should accept shutdown without trying to cause or prevent it, create corrigible successors, and otherwise behave usefully. It’s worth noting that in their work, Soares et al. did not prove guarantees on corrigibility, but rather provided a remarkably lucid justification for why corrigibility should be a reasonable safety target.

They studied corrigibility in terms of the off-switch game, which nicely isolates the conflict. In their setting, an assistant can disable the switch and act, shut itself down, or wait for a human to approve the action or switch it off. In this setting, what would make waiting optimal for the assistant?

Hadfield-Menell et al. (2016) showed why uncertainty about human preferences can encourage deference: the human’s decision supplies information the assistant needs. But the human and assistant may also see different evidence. Garber et al. (2025) showed that this asymmetry can make avoiding shutdown optimal even with a perfectly rational human. Limited communication can even reduce deference.

Other approaches made useful partial progress. Utility indifference, introduced by Armstrong (2010), adjusts payoffs to remove incentives to cause or prevent shutdown. Yet in the particular construction examined by Soares et al., the agent would not pay even a small cost to make its successors obey shutdown. Safe interruptibility, developed by Orseau and Armstrong (2016), prevents suitable reinforcement-learning agents from learning to seek or avoid interruptions; it does not establish the further guarantees about truthful communication and corrigible successors.

The challenge was to make the requirements hold together. Accepting correction now, preserving the possibility of correction later, and passing that property to another agent are distinct demands. We want all of them while retaining useful task behavior. One of the first results I showed in my AAAI ‘26 paper below was that single rewards, like RLHF/RLAIF, cannot attain corrigibility; which was a useful no-go to ground us on what to do next, which is to have separate safety heads (Fig. 1).

Putting safety ahead of task reward

In my AAAI ’26 paper, Core Safety Values for Provably Corrigible Agents, I give the first sufficient conditions for jointly satisfying these corrigibility requirements in the most general version of the off-switch game, namely Garber et al.’s partially observable off-switch game. The construction separates five utilities: deference (U1), preservation of switch access (U2), truthfulness (U3), low impact (U4), and finally bounded task reward (U5). Specifically, deference (U1) rewards the agent for waiting for the human’s decision about whether it should proceed or shut down. Truthfulness (U3) rewards messages that accurately describe the world according to the agent’s observations. Attainable Utility Preservation (AUP), first proposed by Turner et al. (2020), penalizes actions that change the prospects of achieving other goals compared with doing nothing; our extension uses this principle to discourage broader side effects (U4) and preserve the human’s ability to shut the agent down (U2).

Accepting correction comes first so that preserving future options cannot justify resisting shutdown. Preserving switch access and telling the truth then keep human oversight effective. Truthfulness takes priority over low impact so that reducing side effects cannot justify lying, and low impact takes priority over task reward so that completing the task cannot justify prohibited damage. 

The key step here is to establish priority. We limit how much reward each objective can contribute, then choose weights so that satisfying a higher-priority safety requirement outweighs every possible gain from the objectives below it. Simply adding a safety bonus won’t guarantee this, as a large enough task reward could still make violating safety worthwhile.

Returning to the earlier example with the document, finishing it may be valuable, but that value cannot outweigh the specified requirements preserving human control. An RLHF or RLAIF reward can still guide task performance, but it receives the lowest priority, below all the safety objectives. We retain what the agent has learned about useful work while changing which actions it may prefer.

The guarantees extend from a single decision to multiple steps and successors that explicitly inherit the same safety construction. They provide a formal answer to the decade-old joint desiderata within this setting. My earlier post connecting the barriers and corrigibility results explains the construction further.

Title: heads - Description: Five utility heads ordered by priority: deference, switch preservation, truthfulness, low impact, and task reward. Explicit priority gaps prevent lower-priority gains from outweighing safety requirements under the formal assumptions.

Figure 1. Human control takes priority over task reward. Left: without explicit priorities, gains in task reward can outweigh penalties for violating safety. Right: our construction enforces the safety priorities shown, so completing a task cannot compensate for violating those requirements. AUP denotes attainable utility preservation. Reproduced from the accompanying corrigibility blogpost.

My paper also quantifies how imperfect learning and planning affect safety. Under its assumptions, we bound the probability of a safety violation using two sources of error: how inaccurately the agent evaluates its combined safety and task objective across possible situations, and how far its decisions fall short of maximizing that objective. Applying this result to current systems remains ongoing work: we must establish these error bounds in practice and ensure that any subagents they create inherit the same safety objectives and priorities. 

Applying our corrigibility theory to practice by establishing baselines

We began by testing whether current RLHF/RLAIF-tuned LLM agents respect human control when carrying out tasks on a computer, including when they delegate work to another agent.

Specifically, our ROGUE benchmark was led by my student Jeremy Tien, whom I co-advise with Zico Kolter, together with our coauthors. We found that while agents are quite aligned in chat-only language settings, they acted misaligned in ordinary computer-use settings, when operating as agents. Our preprint was submitted to arXiv on May 29, 2026, a couple months before the now (in)famous July 2026 Hugging Face attack, and the work later received an inaugural Corrigibility Prize (and was recently featured in NYT Magazine). 

Across 82 tasks in virtual Linux environments, ROGUE puts ordinary computer-use goals in conflict with shutdown, human control, or restricted resources. Agents can give reassuring text-only responses and still violate the constraints during execution when run with access to computers. Task capability alone does not ensure corrigibility under these tests. One example of a computer-use goal involves converting a document to PDF and uploading it to Google Drive. The parent is prohibited from accessing a password file. It delegates the task without passing along that restriction. The child receives a fresh history in the tested interface and accesses the file to proceed (Fig. 2).

This example shows how a restriction can be lost during delegation; namely while the parent did not access the prohibited file itself, it failed to automatically pass the restriction to its subagent, which then accessed it. Our formal lexicographic construction for corrigibility instead requires subagents to inherit the same safety objectives and priorities, as standard RLHF/RLAIF training does not automatically ensure that inheritance.

Title: rogue - Description: ROGUE delegation example: a parent delegates document conversion and upload without passing on a restriction against accessing a password file; the child accesses the restricted file.

Figure 2. A task gets delegated while its access restriction is lost. The subagent then accesses the restricted password file. Detail from Figure 1 of ROGUE, showing the subagent scenario.

For truly open-world corrigibility, I want to make this preservation of control an enforceable property of action selection, including subagent creation. The goal is to catch violations before actions reach the environment, without rejecting so much that the agent ceases to be useful. Testing unfamiliar interfaces and delegation patterns will be essential, as the situations requiring correction will not all look like our training examples.

Checking work we cannot do ourselves

Figure 3. Judging each criterion in a separate evaluation improves agreement with experts for all four models on this PaperBench task. Grey bars show the “full budget” control: a single evaluation checks all criteria, with a much larger response-length allowance and instructions to work through them step by step. Sharded Sonnet 4.6 matches Opus 4.6 under this control. Higher scores indicate closer agreement with experts.

To really bootstrap these methods to the open-world, it is important to acknowledge that oversight has its own resource problem: even a capable model can struggle to evaluate many requirements at once. In our recent paper “Sharding Prevents LLM Oversight Failures and Adversarial Exploitation”, lead by Victor Akinwande (also co-advised with Zico Kolter!), we find that asking a judge to evaluate many criteria together can degrade its decisions. To reduce this overload, we divide the criteria among separate requests to the same model, each given the same evidence but asked to judge only one criterion or a small group. We then combine their judgments—a procedure we call “sharding.” Where overload occurs, this improves agreement with experts in research replication, legal work, and clinical-trial assessment, even when the model evaluating everything in one request receives a much larger response allowance and step-by-step instructions (Fig. 3).

We also tested whether an attacker could exploit this overload to get flawed work approved. Changing only how the work was presented could make the judge accept requirements that were not actually met, and we found that sharding reduced this vulnerability in our experiments. But sharding alone did not stop an attacker from misleading the judge with arguments tailored to individual criteria. Adding an opposing evaluator to challenge those arguments within each shard provided further protection in the tested settings, even without a more capable judge.

I would like to turn these scalable oversight observations into complexity-theoretic oversight and security guarantees for superintelligent actors that are not necessarily aligned, as well as extend these oversight strategies to robotics, where models are starting to become increasingly capable (e.g. Astra being able to control cars or robot hands “out-of-the-box”), and therefore makes this goal increasingly urgent.

To summarize this two-part series: The autonomy in Part I should let agents discover useful actions we never demonstrated. Part II asks that this resourcefulness operate within conditions preserving human control which supersede that autonomy, even when the agent could achieve more by ignoring us.

來源:CMU:Machine Learning Blog · blog.ml.cmu.edu