BuildProof 對 44 個真實程式錯誤的測試捕捉 32 個
I tested my AI bug hunter on 44 real bugs from Express, ESLint, Hexo and others. It caught 32.
BuildProof 作者以 BugsJS 中 44 個來自 Express、ESLint、Hexo 等專案的真實錯誤測試其工具,捕捉到 32 個。工具會為程式變更生成測試,分別在舊版與新版程式上執行,再由另一個 AI 判斷行為改變是否屬於錯誤。在以實際修復版本進行的反向測試中,44 次有 4 次誤報,單次檢查成本約 $0.15。
AI writes more and more of our code, and code review can't keep up. So I built BuildProof: a tool that looks at a code change and tries to prove a bug, not just guess one.
How it works
- The AI reads the change and suspects a bug.
- It writes a test, and that test runs on the old code AND the new code.
- If the test passes on the old code and fails on the new one, that's a real behaviour change.
- A separate AI "judge" decides whether that change was intended or a bug.
- You only get bugs that come with a failing test as proof.
The test
BugsJS is a public dataset of real bugs from real projects (Express, ESLint, Hexo, Karma, Shields, Bower, hessian.js). I re-introduced each bug and let the hunter look at the change, with no hints.
Results
- Caught 32 of 44 real bugs.
- In the reverse direction (the actual fix, which is correct code), it raised a false alarm 4 times out of 44.
- Cost: about $0.15 per check.
It also knows when to stay quiet: I ran it on 4 open pull requests in lodash, koa and axios. It found nothing, and it was right not to. It doesn't invent bugs to look busy.
Your code runs in a locked container with no network and is deleted when the check ends. There's also a free mode that uses your own local model (Ollama).
Try it
30-second demo, no project needed:
npx buildproof try
On your own git project (3 free cloud checks):
npx buildproof hunt --base main --cloud
I'd really like feedback, especially on the misses. Which kinds of bugs should it catch that it doesn't?
來源:DEV Community · dev.to