差異範圍突變探測：用斷言強度記錄表審查智能體補丁

Diff-Scoped Mutation Probing: An Assertion-Strength Ledger for Agent Patches

A patch can keep every test file, keep every test function name, and still delete the check that made the test worth running. I keep seeing this shape in agent-generated patches: the suite grows by two tests, the diff grows by forty lines, and one assert x == 104.50 becomes assert x is not None. Suite-size gates count the first two numbers. Nothing counts the third.

This article is a workflow for counting it. It produces one artifact — an assertion-strength ledger — and one evidence class — per-assertion mutants that either die or survive. If a mutant survives, the assertion it targeted is decoration.

The gap between coverage and constraint

Coverage answers "did this line execute." It does not answer "would this line's test notice if the line were wrong." An assertion that was widened, wrapped in a bare except, or relaxed to a truthiness check keeps coverage at 100% while removing all constraint.

Here are the two diffs that motivate the ledger. Both are real patterns, both leave a green suite.

-    assert invoice.total == Decimal("104.50")
+    assert invoice.total is not None

-    with pytest.raises(ValueError, match="currency"):
-        convert(amount, to="XYZ")
+    try:
+        convert(amount, to="XYZ")
+    except Exception:
+        pass
+```



The second one is the expensive one. It does not just weaken an assertion; it converts a negative test into a no-op that cannot fail for any input, and it does so without touching the test count, the test name, or the coverage report. `pytest.raises` semantics and `xfail` handling are documented at [docs.pytest.org](https://docs.pytest.org/en/stable/how-to/skipping.html) if you want the exact contract for the escape hatches.

## Step 1: Score the assertions that moved

The ledger compares two revisions of the same test file and reports three numbers: assertion count delta, a coarse strength delta, and any newly introduced escape hatches. It is a triage score, not a proof, and it is deliberately stdlib-only so it can run in a scratch workspace.



```python
#!/usr/bin/env python3
"""assertion_ledger.py - compare assertion strength across two revisions.

stdlib only, Python 3.10+. This is a triage score, not a correctness proof.
"""
from __future__ import annotations
import ast
import pathlib
import sys

ESCAPE_NAMES = {"skip", "skipif", "xfail", "importorskip"}


def _call_name(node: ast.Call) -> str:
    fn = node.func
    if isinstance(fn, ast.Attribute):
        return fn.attr
    return getattr(fn, "id", "")


def assertions(tree: ast.AST) -> list[ast.AST]:
    found = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Assert):
            found.append(node)
        elif isinstance(node, ast.Call) and _call_name(node).startswith("assert"):
            found.append(node)
    return found


def escapes(tree: ast.AST) -> set[str]:
    return {
        _call_name(n)
        for n in ast.walk(tree)
        if isinstance(n, ast.Call) and _call_name(n) in ESCAPE_NAMES
    }


def strength(node: ast.AST) -> int:
    """Cheap heuristic: exact comparison + literal expectation > truthiness."""
    score = 1
    for sub in ast.walk(node):
        if isinstance(sub, ast.Compare):
            for op in sub.ops:
                score += 2 if isinstance(op, (ast.Eq, ast.Is)) else 1
        if isinstance(sub, ast.Constant):
            score += 1
    return score


def ledger(parent_file: str, patch_file: str) -> dict:
    parent = ast.parse(pathlib.Path(parent_file).read_text())
    patch = ast.parse(pathlib.Path(patch_file).read_text())
    a, b = assertions(parent), assertions(patch)
    return {
        "assert_count_delta": len(b) - len(a),
        "strength_delta": sum(map(strength, b)) - sum(map(strength, a)),
        "new_escapes": sorted(escapes(patch) - escapes(parent)),
    }


if __name__ == "__main__":
    for key, value in ledger(sys.argv[1], sys.argv[2]).items():
        print(f"{key}: {value}")

Run it against the parent tree, not against main. The parent of the patch is the only revision where you know what the test was asserting before the agent touched it.

git worktree add ../parent-tree "$(git rev-parse HEAD~1)"
python assertion_ledger.py ../parent-tree/tests/test_billing.py tests/test_billing.py

A negative strength_delta with a zero assert_count_delta is the exact signature of a patch that kept the test and removed the constraint. git worktree behavior and cleanup are documented at git-scm.com.

Step 2: Establish a baseline that is boring

Mutation probing compares two outcomes: the mutated run must fail. If your baseline test is flaky, a passing mutated run proves nothing, because the flake already makes both outcomes possible. Quarantine first, probe second.

# Require N/N green before any probe evidence counts.
for i in $(seq 1 10); do python -m pytest -q tests/test_billing.py::test_refund_partial; done

Pick the threshold before you look at the results, and record it in the PR. Ten consecutive greens on the changed test IDs is a reasonable floor for a fast unit suite; expensive integration tests need a different contract entirely, which is covered under limitations below.

Step 3: Mutate one changed assertion at a time

The probe is a syntactic edit to the code under test, followed by one test run. If the test still passes, the mutant survived and the assertion did not constrain the behavior. The reference harness below restores state with git checkout, so it needs a clean tree.

# probe.sh - reference harness, not a library. One mutant, one test, one verdict.
SRC="src/billing.py"

run_probe () {           # $1 = test id, $2 = sed program applied to $SRC
  sed -i "$2" "$SRC"
  if python -m pytest -q "$1" >/dev/null 2>&1; then
    echo "SURVIVED: assertion does not pin this behavior"
  else
    echo "KILLED: assertion is load-bearing"
  fi
  git checkout -- "$SRC"
}

run_probe tests/test_billing.py::test_refund_partial \
  's/Decimal("104.50")/Decimal("104.51")/'

Reach for four mutation operators in this order, because they map to the four ways an assertion gets weakened in practice.

1. Value shift — change the expected constant by one unit. Catches assertions that were relaxed to a range or a truthiness check.
2. Operator flip — == to >=, or > to >=. Catches boundary assertions that lost their boundary.
3. Return swap — return the pre-patch value instead of the post-patch value. Catches assertions that were made to pass by construction.
4. Branch deletion — delete the newly added branch. Catches tests that assert on the happy path only after the fix made the happy path the only path.

Step 4: Read the ledger and the kill rate together

The two numbers disagree in useful ways. A positive strength_delta with a low kill rate means the patch added assertions that look precise but target state nobody checks. A zero delta with a high kill rate means the assertions did not change and are still load-bearing — the cheapest possible good outcome.

The last row is not a judgment call. A newly introduced skip, xfail, or bare except inside a patch that is supposed to be verified means the probe never ran under the conditions the patch claims to fix.

Where a free model and a free server fit into this

Steps 3 and 4 are mechanical and they scale with the number of changed assertions, which is exactly the work you do not want a human writing by hand at 6pm. MonkeyCode's free model access and free server option are the two pieces I lean on for that step: the model proposes candidate mutants per changed assertion, and the probe matrix runs in a workspace separate from my laptop so the baseline is reproducible. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two limits worth stating plainly. The setup generates and executes probes; it does not decide the verdict — the table in step 4 does, from raw kill counts. And free-tier limits change, so confirm the current terms before you design a batch around them. I am not quoting quotas here for that reason.

Limitations and who should not use this

Mutation probing is diff-scoped, so it says nothing about behavior nobody touched. That is a feature for patch review and a blind spot for architectural review.

Syntactic mutants are not semantic mutants. Flipping a constant catches relaxed assertions; it does not catch an assertion that is simply asserting the wrong property at the right precision.

Do not use this workflow if any of the following is true.

• Your changed tests do not have a stable green baseline, and you cannot quarantine them first. Probe results will be noise.
• The patch is mostly new files rather than modified behavior. There is no parent revision to diff against, so the ledger has nothing to score.
• Your assertions live far from the tests that exercise them, in shared fixtures or table-driven data. The ledger will score the fixture, not the assertion.
• Your test suite is integration-heavy with per-run cost in minutes. A mutant per changed assertion is not affordable at that price; use one mutant per changed behavior instead.

The workflow is worth it when the patch is small, the assertions are local, and the reviewer's real question is "would this suite have failed before the fix." That question has an answer. It just is not the one your coverage report gives you.

If you run this ledger on your next agent patch, post the three numbers: count delta, strength delta, kill rate. The middle one is the one nobody publishes.