跳到正文
DEV Community· billbin1969-debug·· 12 天前AI 評分61

BuildProof 對 44 個真實程式錯誤的測試捕捉 32 個

I tested my AI bug hunter on 44 real bugs from Express, ESLint, Hexo and others. It caught 32.

AI 導讀

BuildProof 作者以 BugsJS 中 44 個來自 Express、ESLint、Hexo 等專案的真實錯誤測試其工具,捕捉到 32 個。工具會為程式變更生成測試,分別在舊版與新版程式上執行,再由另一個 AI 判斷行為改變是否屬於錯誤。在以實際修復版本進行的反向測試中,44 次有 4 次誤報,單次檢查成本約 $0.15。

正文

billbin1969-debug

AI writes more and more of our code, and code review can't keep up. So I built BuildProof: a tool that looks at a code change and tries to prove a bug, not just guess one.

How it works

  1. The AI reads the change and suspects a bug.
  2. It writes a test, and that test runs on the old code AND the new code.
  3. If the test passes on the old code and fails on the new one, that's a real behaviour change.
  4. A separate AI "judge" decides whether that change was intended or a bug.
  5. You only get bugs that come with a failing test as proof.

The test

BugsJS is a public dataset of real bugs from real projects (Express, ESLint, Hexo, Karma, Shields, Bower, hessian.js). I re-introduced each bug and let the hunter look at the change, with no hints.

Results

  • Caught 32 of 44 real bugs.
  • In the reverse direction (the actual fix, which is correct code), it raised a false alarm 4 times out of 44.
  • Cost: about $0.15 per check.

It also knows when to stay quiet: I ran it on 4 open pull requests in lodash, koa and axios. It found nothing, and it was right not to. It doesn't invent bugs to look busy.

Your code runs in a locked container with no network and is deleted when the check ends. There's also a free mode that uses your own local model (Ollama).

Try it

30-second demo, no project needed:

npx buildproof try

On your own git project (3 free cloud checks):

npx buildproof hunt --base main --cloud

I'd really like feedback, especially on the misses. Which kinds of bugs should it catch that it doesn't?

來源:DEV Community · dev.to