以 Diff Coverage 檢查 Agent 補丁的測試是否執行變更行

A green suite after an agent patch tells you one thing: nothing you ran crashed. It does not tell you that anything you ran executed the lines that changed. Those two statements only connect when some test actually enters the new code — and with agent patches, that connection is precisely what breaks.

Diff coverage closes the gap cheaply. Intersect the patch's changed lines with the lines the suite executed, per hunk, and refuse to merge hunks that nobody ran. Below is a working gate for that, plus the one case where the signal lies: a quarantined flaky test.

1. Why file-level coverage cannot answer this

File coverage is an average. A file sitting at 91% can contain a brand-new, entirely unexecuted except branch and the number will not move by a tenth of a point. Agents produce exactly that shape of patch: a small defensive branch, a widened type check, an early return for a case the model imagined but never tested.

Patch-scoped line measurement is the smallest unit that answers the question you actually have at review time: did anything run the lines I am about to approve?

2. Collect the changed line ranges

Three flags matter. -U0 keeps hunks minimal so a single changed line does not drag 40 untouched context lines into your report. --diff-filter=d skips deleted files, which have nothing left to execute. The three-dot range diffs against the merge base, not against whatever main has drifted to since the agent branched.

import re
import subprocess


def changed_ranges(base: str) -> dict[str, list[tuple[int, int]]]:
    diff = subprocess.run(
        ["git", "diff", "-U0", "--no-color", "--diff-filter=d", f"{base}...HEAD"],
        capture_output=True, text=True, check=True,
    ).stdout
    ranges: dict[str, list[tuple[int, int]]] = {}
    path = None
    for line in diff.splitlines():
        if line.startswith("+++ b/"):
            path = line[len("+++ b/"):]
            ranges.setdefault(path, [])
        elif line.startswith("@@") and path:
            m = re.search(r"\+(\d+)(?:,(\d+))?", line)
            start, count = int(m.group(1)), int(m.group(2) or 1)
            if count:                       # pure deletion: nothing to execute
                ranges[path].append((start, start + count - 1))
    return ranges

Pure deletions produce a zero-length hunk and are filtered out. Renames land under the new path, which is what coverage will report. The result is a dict of file -> [(start, end), ...], and it is cheap enough to recompute on every push.

3. Collect the executed lines

One run, branch coverage on, scoped to the source tree:

python -m coverage run --branch --source=src -m pytest -q tests/
python -m coverage json -o coverage.json

coverage json gives you executed_lines and missing_lines per file. Drop --source and the intersection starts paying for conftest.py, fixtures, and test helpers you did not change. Branch coverage is not on by default in every toolchain; if it is off, a half-taken if reports as fully covered.

4. Intersect and classify

Each hunk gets exactly one verdict, and the verdict is what the gate acts on — not the percentage.

import fnmatch
import json
import sys

SEVERITY = ["EXECUTED", "ALLOWED", "UNKNOWN", "PARTIAL", "SILENT"]


def classify(path, span, ran, allowlist):
    lines = set(range(span[0], span[1] + 1))
    if any(fnmatch.fnmatch(f"{path}:{span[0]}", pat) for pat in allowlist):
        return "ALLOWED"
    hit = lines & ran
    if not hit:
        return "SILENT"
    return "EXECUTED" if hit == lines else "PARTIAL"


def main() -> int:
    # args: --base origin/main --coverage coverage.json --allow diffcov.allow
    #       --fail-on PARTIAL --excluded-tests 0
    ran_by_file = {
        f: set(v["executed_lines"])
        for f, v in json.load(open(args.coverage))["files"].items()
    }
    allowlist = [
        ln.strip() for ln in open(args.allow)
        if ln.strip() and not ln.startswith("#")
    ]
    threshold = SEVERITY.index(args.fail_on)
    failed = 0
    for path, spans in changed_ranges(args.base).items():
        ran = ran_by_file.get(path, set())
        for span in spans:
            verdict = classify(path, span, ran, allowlist)
            if verdict == "SILENT" and args.excluded_tests:
                verdict = "UNKNOWN"        # incomplete run, not a clean block
            if SEVERITY.index(verdict) >= threshold:
                failed += 1
            print(f"{verdict:9} {path}:{span[0]}-{span[1]}")
    print(f"\n{failed} hunk(s) at or above {args.fail_on}")
    return 2 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

Illustrative output for a patch that adds a retry branch and never tests it:

EXECUTED  src/ledger/charge.py:41-48
PARTIAL   src/ledger/charge.py:52-61
SILENT    src/ledger/charge.py:64-70

1 hunk(s) at or above PARTIAL

5. Gate it in CI

Severity ordering is the whole point of the enum. --fail-on PARTIAL fails on PARTIAL and SILENT both; --fail-on SILENT fails only on hunks nothing touched. Start strict for one sprint, then decide whether PARTIAL is noise in your codebase or a real signal. Exit code 2 keeps the gate distinguishable from a crashed test run (exit 1) in most CI systems.

python diffcov.py --base origin/main --fail-on PARTIAL --excluded-tests 0

6. What each verdict does not prove

EXECUTED proves a line ran. It does not prove any test asserted on the line's effect. A test that calls a function and checks only that it did not raise will mark every line inside as EXECUTED. Treat the verdict as permission to read the assertions, not as evidence they exist.

PARTIAL is usually the most informative row you will see: a branch taken one way out of two, or a loop body entered by an empty fixture. ALLOWED is a claim, not a fact — each pattern needs an owner and a reason, reviewed like code. Keep patterns line-scoped; src/migrations/* is how an allowlist turns a gate into decoration.

Toolchain limits are real. C extensions report nothing, subprocess code reports nothing, and async task bodies can execute after coverage shuts down. Code that only runs on a real device will read SILENT forever.

7. The quarantine interaction

When a flaky test is frozen out of the run, coverage silently drops. A hunk whose only exercising test was deselected now reads SILENT — the gate is not wrong, the run is incomplete. That is why --excluded-tests downgrades SILENT to UNKNOWN instead of failing: an honest "I don't know" beats a false block that teaches reviewers to bypass the gate.

Pass the count explicitly from your runner, and require a full re-run of the quarantined subset before merge. The opposite failure is worse: never let UNKNOWN become the default by quarantining broadly. If more than a handful of hunks come back UNKNOWN, you do not have a coverage run — you have a habit of skipping one.

8. Seeding the gate, and where MonkeyCode fits

A gate you have never watched fail is decoration. You need a stream of patches — some correct, some with untested branches — to confirm it blocks the right ones and passes the rest. Generating those candidates with a free model tier is a fair use of one: ask for a behaviour change plus tests, then run only the gate and record which verdicts appear. The model is a fixture generator here, not an oracle; a weak sample simply produces an easy EXECUTED row, which is itself a useful calibration point.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option are relevant to the two logistics of this loop: candidate generation and clean-checkout execution. The coverage job must run against a fresh checkout with the same toolchain as CI, on every candidate, or the gate gets skipped exactly when the queue is busy. Running it on a free server is a reasonable way to keep that step from depending on someone's laptop. Two constraints hold regardless of platform: treat the runner as ephemeral (regenerate coverage.json per run, store nothing you cannot reproduce) and pin the toolchain (runtime, coverage, and test runner versions), because a verdict produced by different versions is not comparable to the one your CI will produce. If you want to try the seeding loop, that is a workable starting point — but run the gate locally once first, so you recognise a real SILENT when you see one.

9. Who should not use this

• Repositories where most changed lines are generated code: lockfiles, protobuf output, migrations. The allowlist swallows the gate and you get a report nobody reads.
• Toolchains with no line coverage support for the language under test. Partial instrumentation produces confidently wrong SILENT rows.
• Teams whose flaky rate is high enough that most runs end up partial. Fix the quarantine budget first; this gate amplifies the noise you already have.
• Anyone planning to use the number as a quality metric. It is a merge decision on a diff, not a score for a person or a repository.

The script is under ninety lines and runs in seconds because the expensive part — the test run — was already happening. What changes is that the report now names the exact lines an agent changed and left unexecuted, instead of hiding them inside a green checkmark.