AI 生成測試能否找出程式錯誤？作者以 mutation testing 實測

"AI wrote the tests, they all pass - so quality is guaranteed?" Test counts and coverage cannot answer that, because "the tests pass" and "the tests find bugs" are different properties.

So I let LLMs write pytest suites for three Python projects and compared every suite against the existing human-written suites on one fixed mutant set.

sunnydachs / mutbench

Do AI-written tests actually catch bugs? mutbench compares LLM-generated vs human test suites against one fixed mutant set with mutation testing (paired McNemar, flaky rate, runtime). MIT.

mutbench

Do AI-written tests actually catch bugs? mutbench answers that with mutation testing: it takes a small Python module, has several LLMs (and a human) write a pytest suite for it, then breaks the module on purpose and measures how many breakages each suite notices.

The point is the missing number. A suite can pass 100% of its tests and still assert nothing, and coverage cannot tell the difference. Mutation testing can it changes one thing in the production code and asks whether any test fails.

English | 日本語

What it measures (kept separate on purpose)

…

AI tests are not necessarily weaker than human tests. But "runs", "is correct against the code", and "finds bugs" are three different qualities.

The weak spot turned out to be "the tests don't run": 9 of 27 generation attempts never produced runnable code. And some suites that did run killed zero mutants.

What mutation testing is

Mutation testing plants a deliberate bug in the code and measures whether the tests notice.

The injected changes are things like boundary shifts, flipped comparisons, and altered arithmetic.

def discount(price, rate):
    if price > 10000:      # mutate this to 1000
        rate = rate * 0.8
    return price * (1 - rate)

Run the tests with one mutant in place: if they fail, the mutant is killed; if they pass, it survived.

The three separations

A generated suite never goes straight into a single score:

flowchart TD
    A[generated test suite] --> B{"1. runs at all?"}
    B -->|no| U[UNSCORABLE - never in a score denominator]
    B -->|yes| C{"2. green on clean code?"}
    C -->|no| R[BASELINE_RED - normalized variant, diagnostic only]
    C -->|yes| D{"3. how many mutants does it kill?"}
    D --> E[mutation score + flaky rate + runtime]

Mixing the three makes the numbers lie.

Mutant IDs derive from the target file's path, so every suite faces the same mutant set - that is what makes the per-mutant McNemar comparison possible.

Suites that are red on clean code never enter the ranking table; only their normalized values are reported, as diagnostics.

Results

3 targets x 3 generation paths x 3 seeds = 27 generation attempts:

• Paths A/B/C are three generation routes; the exact model IDs and APIs are recorded in each suite's meta.json in the repo. Path C runs a local model, the other two go through cloud APIs.

• "–" means the seed had zero tests left after normalization, so it could not be scored.

• concise-md is my own small Markdown library; cachetools is a widely used caching library.

Where AI beat the human suite

On concise, path A seed 0 scored 80.0% while the human suite scored 57.1%.

The exact McNemar test over the same 70 mutants gives p=0.0004: of the 20 discordant mutants, the human suite killed only 2 that the candidate missed, while the candidate killed 18 the human suite missed.

In this one case, the AI-written tests found more bugs than the human tests.

But the top seed is not the story: seed 2 of the same path fell back to 57.1%. The three seeds are reported with their spread, not cherry-picked.

Tests that fail on clean code

On keys.py, all three seeds of path A were red on the untouched code:

• seed 0: 8 of 40 tests asserted behaviour the code does not have

• seed 1: all 10 tests did, leaving zero after normalization

• seed 2: 36 of 54 tests did

Asserting behaviour the code does not have is a common failure mode of AI-generated tests.

Dropping those tests silently would hide the generation-quality problem, so the drop counts are stated and the normalized scores are reported next to the originals.

Green but zero

The most striking results of the run:

• path B seed 2 (concise): 2 tests, all 70 mutants survived, 0.0%

• path C seed 1 (concise and full): 1 test, every mutant survived, 0.0%

• full's path C suite took 229 seconds to generate - and still scored 0.0%.

"All tests pass" and "quality is guaranteed" are different claims.

A suite with no assertions collects green results and kills nothing. Those mutants are reported as survivors, never subtracted.

Three seeds, flaky tests, equivalent mutants

The three seeds are independent generations - not the same output scored three times. Flaky rate: every suite ran 20 times under identical conditions, with zero flaky tests - within this environment, these targets, and 20 repeats.

Equivalent mutants (mutations that behave identically to the original) cannot be decided automatically. Mutants that every suite let live are flagged as candidates and reported separately:

concise: 12 / keys: 1 / full: 10

They are never subtracted from the undetected count.

Limitations

• A fixed mutant set is not a sample of real bugs.

• The human baselines are the projects' existing suites, not a controlled baseline.

• Conclusions hold for these 3 targets, 3 paths, and this prompt - no further.

• The statistics are paired per-mutant comparisons; mutants are not independent, so nothing generalizes to "all software" or "all models".

• Execution errors and timeouts are never converted into kills; inconclusive outcomes stay in their own buckets.

Try mutbench

The harness is open source:

pip install -e ".[dev]"
mutbench fetch && mutbench verify
mutbench score targets/concise/concise suites/human-concise --label concise

CI calls no LLM API and needs no secrets; the recorded mutant sets reproduce offline.

If you want to compare your own code's tests against AI-written ones, the repo above has the full recipe.

This is a personal OSS project - no warranty; issues and improvement ideas are welcome.