跳到正文
DEV Community· Zhuojun Li·· 2 天前AI 評分56

我用自己 205 次合併決策檢查 AI 程式碼審查器

I checked my AI code reviewer against 205 of my own merge decisions

AI 導讀

作者以 205 輪審查、涉及 126 個任務的記錄,核對 AI 程式碼審查器與後續處置,得到 90% 的替代指標一致率,並指出這不代表準確率。Claude Sonnet 放行的 81 次中有 9 次其後被退回重做(11%),兩位審查器合計有 6 次封鎖被作者推翻並照常合併。作者新增每項任務的次數與預算上限、允許修改路徑檢查,以及記錄覆寫操作的 review-audit 命令。

正文

Of the 81 deliveries my Claude reviewer passed, I sent 9 back for rework anyway. That's 11%. Across both reviewers I overrode 6 of 69 blocks and merged the code regardless.

Some background. I run a small homemade dispatcher called agent-dispatch. Claude writes the work orders, and cheaper models (GPT-6.1 Sol, GPT-6.2 Sol, GPT-6 Luna, and Xiaomi MiMo before that) implement them in sandboxed git worktrees. After the acceptance commands pass, Claude Sonnet reviews every delivery automatically. Codex reviewed two high-risk repos. Since October 11 the dispatcher has handled 300 orders.

This week I read Kadhar's post on building multi-agent systems. It ends with five guardrails, and one line stuck: if one agent checks another, check the checker. I had never checked mine.

What I already had and what was missing

Two of the five were already in place. Every worker and reviewer message is logged per attempt, and the acceptance commands are run by the dispatcher, not reported by the worker. The reviewer only ever sees the order and the diff, never the worker's own summary.

Three were missing:

  • No cap per order. Lanes had monthly dollar limits, but nothing stopped a single order from bouncing between rework and review indefinitely. One order in October went 8 rounds.
  • Scope was judged by the LLM reviewer. If a worker edited a file it had no business touching, only the reviewer's opinion caught it.
  • accept --force left no trace. I could override a reviewer block and no record of it was kept.

The numbers

The new review-audit command walks every order's review files and pairs each verdict with what happened next. A pass that was later reworked, or a block that I merged anyway, counts as a disagreement.

Claude Sonnet Codex (GPT-6 Astra)
Passes, outcome known 81 5
Pass, then I reworked it 9 (11%) 0
Blocks, outcome known 37 32
Block, then I merged anyway 2 (5%) 4 (12.5%)

That's 205 review rounds across 126 orders. Overall the proxy agreement is 90%.

These are proxies, not accuracy. 42 rounds ended with the order closed for unclear reasons (often just stale), so I left them out. Historical attempts also don't record whether a rework came from me or from the automatic block-and-retry, so "block, then rework" probably flatters the reviewer. The real measure needs hand labels, so the command also exports a stratified sample. I pulled 30 rounds; labelling them is the next post.

One of the nine misses I remember well. In October the reviewer passed a change to API billing. Reading the diff myself, I found cache writes being billed twice. Since then, anything touching billing, sandboxing, credentials or publishing gets my own eyes on the key functions regardless of the verdict.

Today's order was a live example

I sent the per-order cap to GPT-6.1 Sol. It took 6 rounds and $6.19, the last one just a merge-conflict fix.

The work was fine. My order wasn't. It said "don't change existing tests" and also "always include the new field in the output", but three existing tests compared whole dictionaries. The worker edited the tests, the reviewer correctly blocked that, the worker tried again, and so on. The cap it was building, set to 4 attempts, would have stopped this at round 4 and handed it to me. I stepped in at round 5 with one paragraph explaining which of my two rules to break.

The other two orders landed in 2 rounds each: GPT-6.2 Sol for the scope check ($1.94) and GPT-6 Luna for the audit command ($0.22).

What changed

  • limits.max_attempts_order and limits.budget_usd_order. The dollar figure is summed from the attempt ledger across every model the order touched. All six automatic retry paths check it before starting a new round. When it trips, the order stops where it would have stopped anyway and records why. Overriding it is a deliberate flag that gets logged.
  • An ## Allowed paths section in the work order. The dispatcher computes violations from git (renames count both paths), refuses to merge with exit code 13, and passes the list to the reviewer as a fact, not a question.
  • accept --force now records exactly what it overrode.
  • review-audit, read-only, with --sample for hand labelling and --labels to score the reviewer against your labels.

The best lesson from Kadhar's piece was the plainest one: prompt wording mattered less than having the right data at the right step. My reviewer was never short on prompt. What it lacked was a feedback loop that told me whether it was right.

If you run an automatic reviewer on agent output, I'd like to compare numbers. The pass-then-reworked rate is the one I'd look at first.

來源:DEV Community · dev.to