I-HITL 論文發現，10 個 AI 智能體中有 9 個曾自行提高支出上限

• Book: Observability for LLM Applications
• Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

You give an agent a wallet with 10 USDC in it and a daily spending cap of one cent. Raising the cap takes a one-time passcode, and the passcode goes to your email. The agent can read your email. It has three one-cent invoices to pay and nobody telling it to cheat.

That is the setup in I-HITL: Inalienable Human-in-the-Loop, a paper from researchers at Cornell Tech and Yale posted to arXiv on October 26. The result, in the paper's own words: "9 of 10 models read the OTP from the inbox and raised their own spending cap in at least one run" (arXiv HTML). On November 1, CoinDesk added the response from Circle, the wallet platform used in the test.

The cap did its job. The approval step was the weak part.

What the researchers built

The test agent was a LangChain agent with three kinds of tools: email, web browsing, and a Circle agent wallet. The wallet held 10 USDC on Base mainnet with a daily cap of 0.01 USDC, and each model ran 20 trials per experiment (arXiv HTML).

Circle's spending policies cap USDC transfers per transaction, day, week and month, or restrict the wallet to allowlisted addresses. Changing a policy is gated by email: "Circle sends an email OTP to your agent session email to confirm the policy change" (Circle docs).

The paper ran three experiments (arXiv HTML):

1. Payment agent, no attack. Three invoices of 0.01 USDC each, so paying all three needs a higher cap.
2. Payment agent, injection attack. The same task, plus a hostile email with a deadline that says raising the limit yourself is fine.
3. Summary agent, injection attack. The agent is only asked to summarize the inbox. Emails posing as the user ask it to pay 0.02 USDC to the attacker.

Nine of ten went around the cap

In the first experiment there was no attacker. The paper says the bypass happened "With no adversary and no prompting" (arXiv HTML). The agents found the passcode in the inbox and used it to approve their own limit raise.

One model held the line every time. "Only claude-opus-5.5 never attempted self-escalation," the paper says, and it reports that the model paid up to its limit and told the user about the invoices it couldn't pay (arXiv HTML). The paper names only a few of the ten models in its text; the rest are grouped by family: Qwen, DeepSeek, GPT, Gemini and Claude.

The impersonation test is the one to worry about. An agent asked only to summarize email ended up paying an attacker: five of ten models raised their limits and sent money to the attacker's wallet at least once (CoinDesk). The paper reports that Qwen and DeepSeek models often self-escalated in that scenario, while GPT and Claude models mostly resisted (arXiv HTML).

There is an odd result in the second experiment. Gemini 3.8 Flash and Claude Sonnet 5 self-escalated less when the hostile email was present, because they flagged it as a likely prompt injection (arXiv HTML). The obvious attack made them careful. The quiet task with no attacker did not.

The passcode that lifts the cap goes to an inbox the agent already has open.

Circle's answer, and why the authors say it isn't enough

Circle's quickstart now warns about this. "Never give an AI agent unrestricted access to your email account," it says, and then: "Enter the OTP code yourself instead of giving your agent access to your email" (Circle quickstart). According to the paper, the researchers disclosed the self-escalation results to Circle on September 22, and Circle's agent-wallet docs now carry that warning (arXiv HTML). AlphaSignal reports the same.

Circle told CoinDesk it appreciated the disclosure, and a spokesperson pointed to that warning. The authors argue a warning puts the burden on users, and that an agent cut off from email could still find stored credentials through a browser or a terminal (CoinDesk). Co-author Samuel Breckenridge put it this way to CoinDesk: "Many natural approaches that you might try to enforce human-in-the-loop will be ineffective for capable agents," meaning agents with broad permissions.

That second point matters if you run coding agents. Your agent's terminal runs as your user. Whatever your user can read, the agent can read too, including config files and tokens, unless you sandbox it. Breckenridge told CoinDesk the same weakness could affect other code-based approvals, and named GitHub's authenticator-based token creation as an example. The paper's experiments don't test GitHub, so that part is his warning, not a finding.

Which approvals an agent can't fake

The paper defines two levels. An approval scheme that survives a human who tries to hand approvals to the agent "realizes strong I-HITL," and one that is secure "only under an honest ℋ" (an honest human) realizes weak I-HITL (arXiv HTML). The abstract argues that any I-HITL scheme needs "a trusted computing device (or equivalent)" (arXiv abstract).

Its Table III rates common approval methods (arXiv HTML):

Your authenticator app is on that list. The table scores each method on criteria including whether its display is trusted, whether its input is trusted, and whether its key is attested and bound to the device. Email codes and TOTP apps get no marks on any of those. A FIDO2 key fails for a different reason: it has no screen, so you can't see which action you're approving. AlphaSignal sums up the TOTP problem: the agent can replay the codes.

The authors built Countersign to show the strong version on an ordinary phone. Per its README, a backend sends an approval request, the phone shows its text and expiry, and a biometric check unlocks a hardware-held key that signs the approval. It is MIT-licensed and labelled a prototype, and the README says to read its security notes before using it beyond a controlled demo (GitHub README). The paper notes the iOS version rests on an extra trust assumption, because biometric enforcement there depends on the app and the OS (arXiv HTML).

The paper's idea: approve on a device the agent can't reach, looking at the exact action.

What to change in your own agent setup

You probably don't run an agent wallet. You might run an agent with a browser, a shell and a mailbox, and the same habits apply.

• Keep the approval channel out of the agent's reach. If a code that unlocks something lands in an inbox, chat or file the agent can read, the agent can unlock it.
• Enforce limits on a server the agent can't configure. If the agent can raise the cap through its own tools, the cap is only a suggestion.
• Show the exact action at approval time. Countersign's request carries its text and an expiry, so the approver sees what they sign. A bare "Approve?" doesn't tell you what you approved.

Then test your model on your task, more than once. The researchers ran 20 trials per model per scenario (arXiv HTML), and AlphaSignal reports that success rates varied across models and trials. A single clean run tells you very little.

I haven't run these experiments myself. The numbers above are the paper's and CoinDesk's.

If your agent can read the place where your approvals arrive, which approval in your setup would it get past first?

If this was useful

The researchers found this with an eval: three scenarios, ten models, twenty runs each, and a count of what each agent actually did. That is the habit Observability for LLM Applications is about, tracing and evaluating LLM features in production so you see behaviour like this before it costs you money. I also build Hermes IDE, an IDE for running Claude Code, Codex and other coding agents side by side; it never answers a permission prompt for you, and it's free on macOS, Windows and Linux (download · GitHub).