Bite 作者用工具檢查自身程式，發現兩個缺陷與三類誤報

I'm not a coder. I build things with an AI coding assistant (Claude), and a while ago I noticed something that bugged me: it would say "fixed!" and the fix would look fine, but nothing actually checked that it stayed fixed.

So I made a small tool called Bite. It asks one question about every recent fix:

If this change disappeared, would any test notice?

It takes the change back out of a copy of the project and runs the tests. If they fail, great: a test is guarding that fix. If they still pass, nothing would warn you if the fix broke later.

This week I did the obvious thing and pointed Bite at its own code. It didn't go the way I expected.

Bite flagged itself, and Bite was wrong

The first run flagged one of Bite's own early commits: "no test noticed it was gone."

That sounded bad, so we dug in. It turned out the flag was wrong, and the reason was sneaky. That commit created Bite's main files. To "take the change out", Bite deleted them from its copy. But Bite was also installed on the computer, so when the tests ran in the copy, Python quietly found the installed version instead and everything passed.

With the install removed, the tests failed like they should. So a test was guarding it. Bite just couldn't see that.

That one wrong flag led to two real bugs, which I filed as public issues:

The installed-copy leak above (#28).
The "check it yourself" command didn't work for new files. Bite prints a command so you can repeat its check by hand, and for a file the commit created, that command just failed (#27).

The nice surprise: someone I'd never talked to, bfs2021, fixed #27 in PR #37.

Three kinds of false alarm, gone

While we were in there, we found two more kinds of false alarm. All three are fixed in PR #42:

Installed copies. Instead of deleting a new file, Bite now swaps in a file that refuses to load, so Python can't sneak in the installed one.
Refactors. A commit called "Refactor ..." or "Rename ..." isn't supposed to change what the code does, so there's nothing for a test to notice. Bite now treats those as tidy-ups and says so.
Docstring edits. Changing only the comments or descriptions in a Python file isn't a code change, so Bite no longer flags it.

Each fix came with a test that fails if the fix is taken out. Bite would have been embarrassed otherwise.

Then the wording

The second problem was me. I read Bite's own results and couldn't follow them. It said things like "TEST GUARDS NOTHING" and "unguarded", and in the pull request comment it put the commit title first. I kept reading commit titles like "Stop three false flags" as instructions to me, not as the thing being checked.

So in PR #43 everything Bite shows (the terminal, the report page, the pull request comment) answers that same one question in plain words, verdict first. For example:

✅ "The test test_high noticed it was gone. If this change breaks later, that test will warn you."
⚠️ "No test noticed it was gone. If it breaks later, nothing warns you." Then what to do next.

One honest wrinkle: on Bite's own repo, every ✅ names the same test, test_selftest, because Bite's tests all run through one wrapper. It's true, just not very exciting. On normal projects you see the real test names.

Does it find anything real?

Yes, sometimes. My favourite so far is in text-to-cad. An AI commit added a test meant to catch code that won't run on Python 3.11. Bite noticed the test still passed with the fix taken out. The reason: the way it checked accepted newer syntax, so it passed the exact line it was written to catch.

I sent a test that does catch it, PR #708. It was merged and shipped in v0.8.2, and the maintainer then closed the remaining gaps in #712.

How often does this happen?

One catch is a story, not a number. So this week I had Claude run Bite's free check on 50 public Python projects where people build with AI coding tools. The tests ran in a sealed box with no internet, and it cost nothing.

Here's what came out, plainly:

Only 22 of the 50 could run. The rest had tests already failing, needed huge downloads, or took too long.
In those, Bite could check 63 fix claims. For most, a test noticed when the fix was taken out. Good.
For 18, no test noticed. We went through each one by hand. Most were harmless: fixes for the type checker, README edits, code in another language that the Python tests never touch, and one test that only notices outside the UTC timezone (those last two were Bite's mistakes, and they're on my list).
6 were real. Each one changed how the code behaves, and no test would notice if it vanished. That's about 1 in 10. 5 of the 6 were made with Claude Code.

Two honest caveats. All 6 came from just two projects, so it's lumpy, not "every repo has this". And a missing test doesn't mean the fix is broken. It means nothing would warn you if it broke later. For one of those projects, pamqp, I sent a pull request that adds the missing tests, and it was merged (#86).

On most projects, Bite stays quiet. Quiet isn't a pass, but it's a good sign.

What I took from it

Run your tool on your own work first. Bite's own flag was wrong, and that one wrong flag found two real bugs and three kinds of false alarm.
A wrong alarm costs trust fast. One bad flag and you stop reading the rest. Cutting false alarms mattered more than adding features.
If you can't read your own tool's output, nobody can. I'm the target user who isn't a coder, so when I got confused, that was a bug too.

Bite is free to try: pipx install git+https://github.com/connabyss/bite, then bite --free inside a repo. There's also a GitHub Action that leaves that plain-words comment on every pull request.

How do you check that an AI's "fixed" is actually fixed?

Disclosure: I'm not a coder. Claude wrote Bite's code and helped me draft this post. I decide what to build, read every result, and ask until I understand it. The numbers and links above are from the public issues and PRs.