如何在重構前用 coverage.py 找出執行每行變更的測試

Map Every Changed Line to the Test That Ran It Before You Refactor

Start with the uncomfortable result. A repo at high line coverage can still have zero tests touching the function you are about to split. Coverage is aggregated. Refactor risk is local.

Before the smallest safe change, you need a different artifact. Not a percentage. A table that maps each changed line to the tests that executed it.

This post builds that table. It joins a git diff -U0 against coverage.py dynamic contexts. The output is one row per hunk, with a verdict you can gate on.

Why the coverage percentage is the wrong gate

Line coverage answers a repo-wide question. Refactoring asks a line-level one.

Three properties matter for a single changed line:

1. Does any test execute it at all?
2. Which specific tests execute it?
3. How fast are those tests?

A line reached only by a slow integration test is expensive to change. A line reached by three fast unit tests is cheap. The aggregate number hides both cases equally.

Step 1: Record coverage with test attribution

coverage.py can record a dynamic context per executed line. Set it to test_function and every line carries the name of the test that ran it.

# .coveragerc
[run]
dynamic_context = test_function
branch = True

Now run the suite and dump JSON. The --show-contexts flag is what writes per-line test names into the output.

pip install coverage pytest
coverage run -m pytest -q
coverage json --show-contexts -o cov.json

Expect a slowdown. Dynamic contexts cost runtime, so measure it on your own suite. Keep this run serial. Parallel workers scramble the attribution you are trying to collect.

Step 2: Reduce the diff to changed line ranges

You do not need to parse added content. With zero context, the hunk header is the change set.

git diff -U0 --no-color $(git merge-base main HEAD) > change.diff

Headers look like @@ -40,2 +40,3 @@. The +40,3 part means lines 40 through 42 of the new file are touched. A pure deletion gives +40,0, which means there is nothing new to cover.

That is the whole parse. Two regexes, no heuristics, no language model.

Step 3: Join both and print the table

The script below reads cov.json and change.diff. It reports, per hunk, how many tests touched the lines and which lines had none.

#!/usr/bin/env python3
# hunk_gate.py - map diff hunks to the tests that executed those lines.
# Usage: python hunk_gate.py cov.json change.diff  (exit 1 if any changed line has no test)
import json, re, sys
from pathlib import Path

FILE_HDR = re.compile(r'^\+\+\+ b/(?P<path>.+)$')
HUNK = re.compile(r'^@@ -\d+(?:,\d+)? \+(?P<start>\d+)(?:,(?P<count>\d+))? @@')

def changed_ranges(diff_text):
    ranges, path = {}, None
    for raw in diff_text.splitlines():
        if raw.startswith('+++ '):
            m = FILE_HDR.match(raw)
            path = m.group('path') if m else None
            continue
        m = HUNK.match(raw)
        if not (m and path):
            continue
        start = int(m.group('start'))
        count = int(m.group('count') or 1)
        if count == 0:            # pure deletion: no new lines to cover
            continue
        ranges.setdefault(path, []).append((start, start + count - 1))
    return ranges

def lookup(files, path):
    if path in files:
        return files[path]
    for key, value in files.items():   # coverage may store absolute paths
        if key.endswith(path):
            return value
    return None

def main(cov_path, diff_path):
    files = json.loads(Path(cov_path).read_text())['files']
    ranges = changed_ranges(Path(diff_path).read_text())
    print('file'.ljust(38), 'lines'.ljust(10), 'tests'.ljust(5), 'verdict')
    missing_total = 0
    for path, spans in sorted(ranges.items()):
        if not path.endswith('.py'):
            continue
        contexts = (lookup(files, path) or {}).get('contexts', {})
        for lo, hi in spans:
            tests, missing = set(), []
            for line in range(lo, hi + 1):
                ctx = contexts.get(str(line), [])
                tests.update(ctx)
                if not ctx:
                    missing.append(line)
            verdict = 'OK' if not missing else 'NO-TEST ' + str(missing)
            missing_total += len(missing)
            print(path.ljust(38), f'{lo}-{hi}'.ljust(10),
                  str(len(tests)).ljust(5), verdict)
    print('\nchanged lines with no covering test:', missing_total)
    return 1 if missing_total else 0

if __name__ == '__main__':
    sys.exit(main(*sys.argv[1:3]))

Treat this as a sketch, not a shipped tool. Run it on one small module first and check the JSON shape against your installed coverage.py version. The contexts key only appears when --show-contexts was used.

Illustrative output, not from a real repo:

file                                   lines      tests verdict
src/billing/invoice.py                 40-42      3     OK
src/billing/invoice.py                 88-91      0     NO-TEST [88, 89, 90, 91]
src/billing/legacy_discount.py         12-12      1     OK

Step 4: Read column three before column four

A green OK with one test is weaker evidence than NO-TEST on a branch you plan to delete. The line list is the useful part. It is the exact set of lines you are changing blind.

Classify every hunk before you edit anything:

Most legacy modules produce the second row. That is the real finding, and it is invisible in any coverage badge.

Step 5: Run only the attributed tests while you edit

Contexts come back as strings like test_invoice.py::test_proration_rounding. Convert them to node IDs and the loop stays seconds long.

pytest -q src/billing/test_invoice.py::test_proration_rounding \
         src/billing/test_invoice.py::test_zero_quantity

Then change one hunk. Rerun that subset. Regenerate the diff. Repeat. Never batch five hunks and rerun once.

Where a free model and a free server genuinely help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two parts of this workflow are tedious rather than hard, which makes them reasonable candidates for a free tier.

The attributed baseline run. Dynamic contexts force a serial run, so the suite is slower than usual. That is a good job for a remote box. MonkeyCode offers a free server option; I am not asserting a machine size, a time limit, or permanence. If your baseline fits, run the instrumented suite there and pull back cov.json and the diff.

The uncovered-hunk backlog. After the gate, you have a list of ranges nothing executes. Free model access through MonkeyCode is useful for drafting scenarios for those ranges, such as which inputs could reach line 88. You still write the assertions. Delete any proposal that only restates the code in English.

That is the whole product role. Remove both mentions and the gate still works.

Limitations and who should skip this

• Lines, not branches. Contexts attach to executed lines. A covered if body says nothing about the branch you are about to add.
• Covered is not asserted. This gate reports who runs a line. It does not report whether that test would fail on a bad edit. Pair it with a mutation or kill-rate check.
• Subprocess coverage is invisible. If tests drive a server in another process, instrument that process or the application lines will all look uncovered.
• C extensions and generated code. Attribution is unreliable or meaningless there. Omit those paths in .coveragerc.
• Parallel runners. Dynamic contexts plus parallel workers yield attribution you cannot trust. Keep the gate run serial.

Skip this entirely if your suite is integration-only over HTTP. Skip it if a serial run fits no machine you can reach. Skip it if the target code is untested and about to be deleted, because deletion needs no coverage map.

Checklist before you touch the code

1. Set dynamic_context = test_function in .coveragerc.
2. Run the suite serially, then coverage json --show-contexts.
3. Diff against the merge base, not against your last commit.
4. Run the hunk gate and read every NO-TEST line out loud.
5. Classify each hunk with the table above.
6. Add the missing pins first, then make the smallest safe change.
7. Re-run the gate after each hunk, not after the whole refactor.

The map is cheap to build and hard to argue with. It replaces "the suite is green" with "these five tests ran the four lines I am changing." That is a claim you can review before the diff exists.

If you try it on a legacy module, the output that surprises people is usually one slow test covering forty lines. Share the hunk that surprised you.