審批閘門：如何設計不會闖出大禍的智能體

Every agent demo ends the same way: the model does something impressive, then someone in the comments asks "what stops it from deleting the database?" Usually the answer is a shrug.

If you actually run an agent on your own machine, that question stops being rhetorical. Here is the model I landed on after a month of letting one drive a real desktop, and the parts that mattered.

Three tiers, not one switch

The worst design is a boolean: the agent can or cannot act. What you want is tiers.

Tier 0 - reversible and cheap. Read files, run ls, open a URL, take a screenshot, query an API with GET. No approval. Slowing these down destroys the value of having an agent at all.

Tier 1 - reversible but expensive. Write files inside the workspace, install a package, run a build, restart a local service. Pre-approved inside a scope: the workspace directory, this repo, this VM. The gate is geographic, not per-action.

Tier 2 - irreversible or outward-facing. Send an email, publish a post, push to a remote, delete anything outside the workspace, spend money, message a human. Always an explicit yes, every time, with the exact payload shown.

The rule that makes this work: the tier is a property of the action, not of the agent's confidence. A confident agent is not a safer agent.

Show the payload, not the intent

"Approve: send follow-up emails" is useless. "Approve: 3 emails, to these addresses, with this text" is reviewable.

Every approval request should render the concrete diff: the exact command, the exact bytes, the exact recipients. If the UI can't show it, the action is not ready to be approved.

Dry runs are a feature, not a debug mode

--dry-run on anything that touches state. The agent runs the full plan, prints what it would do, exits without doing it. I use this constantly for shell-heavy tasks, and it catches a surprising number of "correct plan, wrong path" errors.

Undo beats prevention

You will never enumerate every dangerous command. So make the common damage reversible: snapshot before editing, git commit before bulk changes, copy instead of move, soft-delete into a trash folder with a timestamp. Then the approval gate only has to catch the things you genuinely cannot undo.

Log the refusals

The most useful rows in my logs are the actions the agent proposed and I rejected. They are a free specification of where the policy is wrong - usually not "too strict", more often "this tier should not exist".

That is the whole design. Tiers by reversibility, payload-level approval, dry runs by default, undo where you can, and a log of every refusal.

If you are building something in this space, I would like to know which tier boundary bit you first - for me it was "install a package", which is reversible in theory and never in practice.