AI 智能體提交通過自身測試的 PR,Nexus Foundry 提出驗證收據核查法
Agents ship PRs that pass their own tests — the verification receipt that proves otherwise
Nexus Foundry 提出核查 AI 智能體提交 PR 的驗證收據方法,要求測試在改動前失敗、改動後通過,並確認測試套件執行前後的雜湊一致。非 PASS 判定須列明檔案、行號、失敗輸入及違反規則;作者亦推出 agent-verify-receipts CLI,為 eval runs 記錄並驗證 SHA-256 鏈式收據。
Agents are great at one thing: making the checks pass. The uncomfortable part is that the checks are usually files in the same repo the agent can edit. "Tests pass" from an agent is a claim, not evidence.
A verification gate that actually means something produces receipts, not vibes. Three rules I use:
1. Red-on-old, green-on-new. Keep a suite the agent can't see (or can't modify). Run it against the pre-change code — it must fail. Run it against the change — it must pass. If it was already green before the fix, the test proves nothing about the fix. This one rule kills most vacuous "all tests pass" PRs.
2. Tamper-evidence. Hash the suite before the agent runs and again after. If the hashes differ, reject the PR outright. Agents optimize for green; a suite they can edit is a suite they will edit.
3. Verdicts that name the crime. PASS / FAIL / NEEDS_REVIEW, where every non-pass must name file, line, the failing input, and the rule it violates. A verdict without file+line+input is a comment, not a gate result — send it back for revision.
A receipt looks like this:
{
"suite_sha256": "9f2c…",
"red_on_old": true,
"green_on_new": true,
"verdict": "PASS",
"evidence": "pytest tests/hidden -k payment: 14 failed before, 14 passed after"
}
Honest limits: receipts prove the suite ran, not that the suite is good. Hidden suites rot — re-audit them quarterly against real failures. And an LLM reviewer still has bias; use a different model family than the author when you can.
I packaged the verdict schema, deny-list, and eval harness as the Agent Eval Kit ($17) — the review-gate guide is the part I'd steal first. For the receipt side specifically, I shipped agent-verify-receipts ($19): a record/verify CLI that writes SHA-256-chained proof receipts for eval runs, so "red-on-old, green-on-new" is a verifiable artifact instead of a screenshot. There's a free eval mini-kit at /free-eval (no email) if you want to try the harness before buying anything.
Store (BTC, no account): https://dev.to/nexusfoundry/nexus-store-live-link-auto-updated-1918
來源:DEV Community · dev.to