讓 Agent 接觸真實檔案前，我會做的兩小時審查

Last spring I let an agent run unsupervised in a real project directory for the first time. It finished the task I gave it, then kept making changes it thought were helpful, and one of those changes overwrote a config file I had spent a week tuning. Nothing was deleted. Nothing looked broken until the next morning. I lost two days rebuilding numbers I had never written down. That was the cheap version of the story.

So now there is a two hour audit before any agent's first real run. Not because I assume it is hostile or incompetent. Because the first unattended run is the only run where I genuinely do not know the shape of the damage, and the price of learning it by accident is a config file, a database index, or a directory of half-written documents. The audit is deliberately boring. Four lists, one switch, one copy, and a condition I can check with a command.

Hour one: map the blast radius

I open a scratch note and write four headings: reads, writes, executes, sends. Then I walk the configuration, the tool definitions, the permission flags, the prompt, and the directory it will live in, and I put every capability I can find under one of the four.

Reads is always the longest list and the one I used to skip. Everything in the working directory, obviously, but also the home directory if the process runs as me, environment variables, SSH keys, browser profiles, a shared mount with client documents on it. I check with a file tree listing taken from inside the sandbox the agent will actually run in, not from my own shell, because those are frequently different worlds. If the tree holds paths I would not paste into a chat window, the agent should not have read access either.

Writes is where the audit earns its keep. A directory, a log file, a temp folder, a package cache, the stdout of a build. For each, I ask what a wrong write costs me. Overwritten notes I can restore. An overwritten migration file can take a whole evening. A write into a directory that syncs to a remote gets copied everywhere before I notice.

Executes covers shell access, package managers, anything that can spawn a subprocess. The question is not whether it can run commands but which commands, and I try to make that list finite. A single easy escape hatch like unrestricted shell turns the other three lists into decoration, because anything the agent can write it can then run.

Sends is the list people forget, and it is the one that has surprised me every time. Domains it can reach, a webhook, a git remote, an email address. I write every destination as a hostname, then turn the list into an allowlist in the actual configuration, so anything not written down fails closed instead of succeeding quietly at three in the morning.

Hour two: build a dry run

A dry run is not a log level or a preview screen. It is a mode where the agent goes through every step except the last one and prints what it would have done. The one I build for a new agent takes about forty minutes and looks like this.

$ agent run --dry-run --task "rebuild the docs index"
[dry-run] would exec : python tools/reindex.py --root ./docs
[dry-run] would write: ./docs/index.json          (2.1 MB, 1 file)
[dry-run] would delete: ./docs/index.json.bak     (regenerate instead)
[dry-run] would send : https://hooks.example.net  -> blocked, not in allowlist
[dry-run] result: 0 files changed, 0 bytes sent, 1 blocked action

The flag is worth having only because it names the exact command and the exact paths. If the output says "would update the index" I go back and make it print the file list, because a summary cannot be checked and a list can. It also has to say what it would have deleted, since deletion is what I am least able to undo.

Then I run the three tasks I actually care about in dry-run mode, in the order I would run them for real, including one I expect to fail. That third one matters most. A dry run that only exercises the happy path tells me nothing about how the agent behaves when it hits a permission error or an empty directory, which is exactly when it improvises something I did not authorize.

The snapshot

Before the first real run, I copy the one directory the agent will write to. Just the directory that appeared in the writes list from hour one, not a full image. I timestamp the copy with the date and the task name so that in six weeks I still know what it is without opening it. Somewhere in the same note I write the restore command in full, and I run it once against a throwaway file so I know the syntax is right when I am tired. A backup I have never restored from is a hope, not a backup.

The finish condition

For each task I intend to run, I write one line that a machine could check. Not "the docs are updated" but "docs/index.json exists, parses as JSON, and contains at least as many entries as the source directory". Not "the tests pass" as an aspiration but as the literal command and its exit code. For anything message-shaped, the condition is a line appearing in an outbox file, so I can verify without relying on the agent's own report.

If I cannot write that line, the task is not ready. That rule has killed more tasks than any other part of this audit, and it has been right nearly every time. A task without a checkable finish condition is a task where the agent's own summary becomes the ground truth, which means I am reading a description of the work instead of the work.

The first real run

When the real run happens, I watch it. In the room, with the log streaming next to me. I keep that log, dated, beside the snapshot, because in an hour it will be the only record of what happened and the diff alone will not explain why.

Afterwards I read the diff line by line. The summary at the top I ignore until the end, and sometimes I read the diff a second time backwards, starting from the files I care most about. Summaries are written by the same process I am auditing. They are useful for orientation and useless as evidence. Reading the diff takes fifteen minutes on a small task, and those fifteen minutes are what catch the change nobody asked for.

What this actually buys you

I want to be honest about the size of the thing. Two hours of audit does not make an agent safe. It does not prevent the confident wrong edit, the plausible file with wrong contents, the permission that turns out broader than the documentation said. A checklist fixes none of that.

What it buys is a different distribution of failures. Without the audit, the first failure is large, silent, and discovered by someone else. With it, the failure tends to be small, loud, and mine. A blocked domain in a dry run, a diff line I did not expect, a restore that takes four minutes instead of two days. The difference is who finds out, when, and whether the record of what happened is sitting in a log file or in my memory at nine in the morning.

That is the whole audit. It fits on one page, it takes two hours, and it is the cheapest insurance I have found for the specific moment when a very confident process gets write access to files I care about.