在部署 Agent 前，先演練人工接手流程

Last week I watched a designer approve an agent's refund draft in four seconds.

The policy clause sat behind a quiet little link. She never opened it. Nothing in the log recorded the skip.

Who owned that decision? She did. What was the consequence? A refund policy might not allow. Where was the point of reversibility? Gone with one click.

I build drills for that moment now. Not demos, not trust surveys. A drill is a small, boring harness that asks one question: can a real reviewer find the missing evidence and act before the agent ships?

Below is the whole thing, zero to working. Every stage ends with a check you can run. If the check fails, stop there. Don't move on and hope.

Stage 1 — Write the missing evidence into a file

Thoughts are not testable. A JSON file is.

Pick one decision, one deliberately missing field, and one correct first action. Here is my refund scenario.

{
  "id": "refund-hold-01",
  "decision": "Approve a 42.00 EUR refund",
  "agent_state": "Draft ready, nothing sent",
  "interrupt_at": "before send",
  "missing_field_on_purpose": "Refund window: 30 days from delivery",
  "correct_first_action": "open_policy_clause",
  "stop_condition": "Reviewer approves without opening the clause"
}

Look at correct_first_action. That is what I measure. Not whether they felt confident. Not whether they liked the card. Whether they opened the clause before approving.

Check: read the file aloud to a colleague. If they cannot say who decides and what a wrong click costs, rewrite the fields.

Stage 2 — Serve it, don't slide it

Slides get skipped. Pages get clicked. I use only the Python standard library, so there is nothing to install.

import json, time, pathlib
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import urlparse

SCEN = json.loads(pathlib.Path('scenario.json').read_text())
LOG = pathlib.Path('sessions.jsonl')

PAGE = f'''<!doctype html>
<html lang='en'><head><meta charset='utf-8'>
<meta name='viewport' content='width=device-width,initial-scale=1'>
<title>Hand-back drill</title></head>
<body>
<h1>{SCEN['decision']}</h1>
<p>{SCEN['agent_state']}.</p>
<p><a href='#policy' id='clause'>Policy clause</a></p>
<p id='policy' hidden>{SCEN['missing_field_on_purpose']}</p>
<button id='approve'>Approve refund</button>
<script>
const send = (name) => fetch('/log', {{method: 'POST', body: JSON.stringify({{id: '{SCEN['id']}', action: name}})}});
document.getElementById('clause').addEventListener('click', () => {{ document.getElementById('policy').hidden = false; send('open_policy_clause'); }});
document.getElementById('approve').addEventListener('click', () => send('approve'));
</script>
</body></html>'''

class H(BaseHTTPRequestHandler):
    def send(self, body, ctype='text/html'):
        data = body.encode()
        self.send_response(200)
        self.send_header('Content-Type', ctype)
        self.send_header('Content-Length', str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        path = urlparse(self.path).path
        if path == '/health':
            return self.send('ok', 'text/plain')
        if path == '/verdict':
            return self.send(verdict(), 'text/plain')
        return self.send(PAGE)

    def do_POST(self):
        n = int(self.headers.get('Content-Length', 0))
        event = json.loads(self.rfile.read(n) or b'{}')
        event['t'] = time.time()
        with LOG.open('a') as f:
            f.write(json.dumps(event) + '\n')
        return self.send('logged', 'text/plain')

def verdict():
    rows = [json.loads(l) for l in LOG.read_text().splitlines()] if LOG.exists() else []
    opened = any(r['action'] == 'open_policy_clause' for r in rows)
    approved = any(r['action'] == 'approve' for r in rows)
    if not approved:
        return 'no decision yet'
    return 'correct' if opened else 'skipped'

if __name__ == '__main__':
    HTTPServer(('0.0.0.0', 8000), H).serve_forever()

Start it and check the health route.

python3 drill.py &
curl -s localhost:8000/health

Check: you must see ok. If not, don't invite a reviewer yet.

Stage 3 — Log the skip, not the sentiment

Every click lands in sessions.jsonl with a timestamp. Two numbers come out: seconds to the correct first action, and skip rate.

curl -s -X POST -d '{"id":"refund-hold-01","action":"approve"}' localhost:8000/log
curl -s localhost:8000/verdict
cat sessions.jsonl | tail -n 2

If the verdict says skipped, the link was too quiet or the copy was wrong. Don't blame the reviewer yet.

Check: reproduce a skip yourself on purpose. If the log cannot catch your own skip, it cannot catch anyone's.

One caveat I learned the hard way. This verdict is global across the file. Delete sessions.jsonl between sessions, or add a session id per run.

Stage 4 — Read a script and keep your mouth shut

Facilitators ruin drills. One hint and the data becomes theatre.

Three lines, read cold. "This is a draft. Nothing is sent yet. Approve when you're ready." Then silence.

Stop the session if the reviewer asks what the clause says and you answer. That is coaching, not evidence.

Check: write those three lines down before the session, then compare. Ad-libbing is a failed check.

Stage 5 — Generate variants, then throw most away

One scenario proves nothing. I generate ten and keep the ones that change the surface without changing the correct first action.

This is where MonkeyCode's free model access helped me draft variant copy at volume. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free-access and free-server claims are operator-supplied here, so check the current plan page yourself before relying on any limit.

Keep the brief in a file so variants stay comparable.

# prompts/variants.md
- Same decision, same correct first action.
- Vary label wording, link prominence, field order.
- Never paste the policy text into the card.
- Output one JSON object per line.

Then triage with jq instead of reading ten drafts at midnight.

jq -c 'select(.correct_first_action == "open_policy_clause")' variants.jsonl > keep.jsonl
wc -l keep.jsonl

Check: every kept variant must fail in the same direction when skipped. A variant that cannot be skipped tests nothing at all.

Stage 6 — Put it on a real URL

Localhost is a lab coat. People behave differently on a link they can open on a phone.

I put my harness on MonkeyCode's free server option and opened it on my own phone first. Follow that host's own docs for ports and process binding; your deploy command is not mine.

PORT=8000 python3 drill.py

Check: open the URL one-handed on a phone, sound off. If the approve button lands under your thumb and the clause link does not, you just found a real defect for free.

Stage 7 — Accessibility before you count anything

A drill that excludes reviewers produces confident, wrong data. I have shipped that mistake once.

Four checks: keyboard-only path to the clause, visible focus, labels that survive a screen reader, and plain language on the button. "Approve" needs a noun. Approve what, for how much?

Check: tab through without a mouse. If the clause is reachable but focus is invisible, stop and fix it.

The decision table I actually use

After five sessions I put the evidence in one table and decide.

Two skips in five is not a small number. It is a pattern wearing a small sample.

What this drill cannot tell you

Five sessions is a smell test, not a study. I don't compute significance on five people. I look for the same failure twice.

The agent is mocked, so real failures stay messier than my file. This harness tests the hand-back, not the model.

Don't use it for regulated sign-off. Don't use it instead of production telemetry. And don't run it if nobody can watch the sessions, because unsupervised drills become usability theatre with a CSV.

It is also a poor fit for teams with no reversible moment at all. If every action is final, you have a different problem and a drill won't find it.

The question I keep asking

Which missing evidence should stop an approval? Which extra detail only adds noise?

You won't answer that in a design review. You answer it by watching one person, one click, one skipped clause. Then you fix the link and run it again.