如何構建能識別失效基準的評測：智能體基準審計實驗
Prove It Works: Building an Eval That Can Catch a Broken Benchmark
作者建立兩套各有 60 項任務的智能體基準評測，一套乾淨，另一套植入 12 處測試缺陷；審計找出 11 處，沒有誤報。參考智能體在乾淨套件得 60/60、缺陷套件得 55/60，測試本身令分數損失 8.3 分；三個評分器在審計前以 150 個已標註項目驗證。
A benchmark with flawed tests measures the tests. So I built two suites — one clean, one with 12 flaws planted in the tests — and watched a do-nothing agent start scoring points while my reference agent lost 8.3 points to the tests themselves.
This is the full experiment: calibrating the graders first, pre-registering the audit bar, catching 11 of 12 flaws — and being honest about the one that got away.
The setup
Local, deterministic, no API keys — a lab for auditing your eval before your eval audits your agent. The setup is a clean 60-task suite (arithmetic, extraction and classification; gold answers cross-checked by an independent second implementation) and an identical twin with 12 flaws planted in the tests.
What the flaws did
- Null agent (outputs nothing): 0/60 on the clean suite, 2/60 on the flawed suite
- Echo agent: 0/60 on the clean suite, 4/60 on the flawed suite
- Reference agent: 60/60 on the clean suite, 55/60 on the flawed suite — 8.3 points lost to the tests, not the tasks
Calibrate the graders first
Three graders were validated on 150 labelled items before any audit run: exact-match 80.0%, normalized 100%, rubric 99.3%. The point is stated plainly in the lab: an uncalibrated grader is just another flawed test.
The audit
The audit bar was registered before the run: at least 10 of the 12 flaws caught, at most 3 false flags. The audit caught 11 of 12 planted flaws with zero false flags. The one miss, flaw-09, had its gold and its source document disagreeing (300 vs 315) from the same table — shared-source flaws are the hard class, and that is exactly what to test for next.
If you build or trust agent benchmarks, audit the proof itself, not just the score.
來源：DEV Community · dev.to