Keerthirajan Senthilkumar 開發 bugbout，首次測試發現 66 個編碼智能體補丁中有 27 個錯誤

Keerthirajan Senthilkumar

Every coding agent I've used ends its work the same way. A tidy summary, a list of files it changed, and some version of "the issue is resolved." It's convincing. It's also something you mostly have to take on trust, unless you sit down and review every patch yourself, which defeats a good part of the point.

I wanted to know how often that trust is deserved. Not on a benchmark, on real code. The answer from my first proper run: agents handed back 66 patches as done, and 27 of them were wrong.

This post is about how I measured that, what I found when I tried to get agents to catch each other's mistakes, and the open source tool that came out of it, bugbout.

Benchmarks answer a different question

There are good public benchmarks for coding agents. The best known is SWE-bench, which collects real GitHub issues from popular Python projects and checks whether an agent's patch passes the tests the maintainers added with their own fix.

That method is the right one. The problem is the dataset. A score on Django and scikit-learn tells you very little about your codebase, with your conventions, your weird internal framework and your test suite. It's also a fixed, public set of problems, which is exactly the kind of thing that ends up in training data.

What I wanted was the same method pointed at my own repository.

Your git history is already a benchmark

Here's the observation the whole tool rests on. Every time someone on your team fixed a bug and added a test for it in the same change, they created a perfect evaluation task without meaning to:

The code just before the fix is the starting point.
The issue they linked describes the problem in a human's words.
The tests they added define what "fixed" means.

Run those tests before and after the real fix. The ones that flip from failing to passing must pass on any correct patch. The ones that passed both times must keep passing, so you catch fixes that break something else. Run the fixed version twice and throw out any test that changes its mind, because flaky tests shouldn't decide anything.

bugbout mines your history for these, builds that ground truth, and then hands each task to every agent you want to compare.

Making it fair was most of the work

The tricky part isn't running agents. It's making sure none of them can see the answer, because every one of these bugs has a public fix sitting on GitHub.

So each agent gets a copy of the code with no .git folder at all. The fix and every later commit are physically absent, not just off limits. The fix's own test files are removed too. Web tools are switched off for every agent. Pull request descriptions often explain the fix in detail, so any section that does that gets stripped from the issue text before the agent sees it.

Agents also run inside an operating system sandbox, seatbelt on macOS and bubblewrap on Linux, where they can read anything but only write inside their own checkout. Their shell commands get no network. When bugbout later runs tests that an agent wrote, those run sandboxed too, and they never see your API keys.

And bugbout never asks an agent what it did. It compares the folder before and after, and that difference is the patch. Whatever the summary says doesn't count.

The first idea didn't work

I want to mention this because it shaped everything after. Before bugbout, I spent a while on a different question: would an AI reviewer catch more injected bugs if it could search the repository's history first?

It didn't. Without the history it caught 52.4% of them, with it 48.8%. When I dug into why, the retrieval was finding the right documents almost every time. The agent just got more ambitious with more context and wrote tests that were wrong about how the library actually behaved.

That result was annoying, but it taught me to measure before believing, to freeze the rules before looking at the final numbers, and to report what actually happened. I kept all three habits for this project.

Can agents check each other's work?

Once you have several agents attempting the same bug, an obvious idea follows. Maybe you don't need the hidden tests at all. Maybe the agents can vote.

So I set up a cross-examination. Every agent fixes the bug and writes a regression test. Optionally, other agents act as examiners: they never see any patch and only write tests from the bug report. Then every test runs against every patch, and a set of plain rules (no AI involved) picks the patch the independent evidence supports. A test only counts as evidence if it fails on the original buggy code, nobody's test is allowed to vouch for their own patch, and when the evidence is thin the judge abstains instead of guessing.

I tuned all of that on 22 bugs from a few projects, froze the rules, and then ran it once on 30 bugs from sqlparse and tomlkit that I hadn't looked at.

Here's what came out.

The best single agent fixed 20 of 30. Picking by cross-examination got 21. The ceiling, where at least one agent was right, was 22. So it helps, but only a little, especially when one agent is already strong.

As a merge gate it was cautious and mostly right. Of the patches it approved, 70% were correct, against 67% if you approved everything. It rejected 3 of the 20 wrong patches and didn't reject a single one of the 40 correct ones.

The more interesting result was which evidence actually tells right from wrong. I measured it with an AUC, which in plain terms is: take one correct patch and one wrong one at random, how often does this evidence rank the correct one higher? 0.5 is a coin flip.

An agent's own regression test, run against other agents' patches: 0.77
Tests from examiners asked for the general rule behind the bug: 0.66
Tests from examiners asked to reproduce the bug: 0.53 to 0.55
The project's existing test suite: 0.45

That last one surprised me most. The tests you already have are nearly useless here, because a wrong fix rarely breaks old behaviour. And tests written from the bug report alone are close to a coin flip, because a wrong fix usually does fix the symptom someone reported. It breaks the case next to it, which nobody thought to write down.

What bugbout is now

The study turned into a tool anyone can run:

pip install bugbout

cd your-project
bugbout init      # reads your layout and the agents you have installed
bugbout doctor    # checks everything before an agent spends any quota
bugbout run       # finds the bugs, races the agents, judges and scores
bugbout report --format markdown

It supports Codex, Claude Code, Gemini CLI and mini-swe-agent out of the box. mini-swe-agent works with any model through LiteLLM, including local ones, so you can compare models as well as products. Anything else with a command line can join through a few lines of config, and other packages can register their own agents without touching bugbout.

A few details I'm happy with. Rate limits shape everything, so runs are queued per quota: your ChatGPT plan, an API key and a local model each go at their own pace without blocking each other. Every finished run is saved as it completes, so you can stop at any point, run again, and nothing is repeated. Add a new agent later and only that agent runs. When an agent hits its usage limit, that run is set aside rather than counted against it.

To check it on something real before calling it done, I ran it on a fresh clone of sqlparse. It found three recent bugs with tests that prove them. Claude Code fixed all three. Codex fixed two, and its third patch looked fine but left the bug in place. Cross-examination picked a correct patch for every one of the three. That's the kind of answer I wanted from the start: specific to one codebase, and easy to read.

What it doesn't do yet

It's version 0.1, so a few honest limits. It works on Python projects tested with pytest, reads issues from GitHub, and only uses fixes that came with a new test. It runs on macOS and Linux, or Windows through WSL2. And it spends your agents' quota, because it really runs them.

Try it

The code is at github.com/Keerthirajan58/bugbout under the MIT license. If you run it on your own repository, I'd really like to hear which agent came out on top, and what surprised you. Issues and pull requests are very welcome, especially adapters for agents I haven't covered yet.