重構前先測試安全網：以變異測試驗證 characterization pins

A green characterization suite is not evidence of safety. It is a claim. Verify the claim before you cut anything.

Most refactors fail in the same way. The pins pass before the change. The pins pass after the change. Then a downstream job breaks, because the pins never actually watched the behavior you moved. You need a score for the net itself.

This post covers one measurable gate: mutation-check the pins first, then make the smallest safe change. Earlier posts in this series built the pins. This one asks whether they can fail.

Two ways a pin suite lies to you

The blind pin. It asserts something stable and irrelevant. assert result is not None passes forever, including after you delete half the report.

The tautological pin. It copies the implementation into the test. It imports private helpers and asserts the same call order the source already has. It fails when you rename, not when you break.

Both produce green CI. Neither produces a safety net. Coverage numbers will not tell you which one you have.

The artifact: a pin sensitivity score

This harness applies small text-level mutations to the source, then runs the pin suite against each mutant. A good net catches most mutants. A decorative net catches none.

Save as pin_check.py. Exit code 0 means the score cleared your threshold.

#!/usr/bin/env python3
"""Measure characterization-pin sensitivity with text-level mutations.

Usage:
  python pin_check.py --repo . --min-score 0.6 \
    --pin-cmd "python -m pytest -q tests/pins"

Exit codes: 0 = score >= min-score, 1 = below threshold, 2 = harness error.
"""
from __future__ import annotations

import argparse
import json
import shutil
import subprocess
import sys
import tempfile
from dataclasses import dataclass, asdict
from pathlib import Path


@dataclass(frozen=True)
class Mutation:
    name: str
    rel_path: str
    old: str
    new: str


# One row per behavior you believe the pins protect.
# Text-level on purpose: cheap, reviewable, no AST tooling required.
MUTATIONS: tuple[Mutation, ...] = (
    Mutation("currency_default", "billing/invoice.py",
             'currency: str = "USD"', 'currency: str = "EUR"'),
    Mutation("round_precision", "billing/invoice.py",
             "round(total, 2)", "round(total, 1)"),
    Mutation("skip_none_rows", "billing/report.py",
             "if row is None:", "if not row:"),
    Mutation("sort_key", "billing/report.py",
             "key=lambda r: r.date", "key=lambda r: r.amount"),
    Mutation("header_order", "billing/report.py",
             '["id", "date", "amount"]', '["date", "id", "amount"]'),
)

IGNORE = shutil.ignore_patterns(".git", "__pycache__", "*.pyc", ".venv")


def run(cmd: list[str], cwd: Path) -> int:
    proc = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)
    return proc.returncode


def apply_mutation(root: Path, mut: Mutation) -> bool:
    target = root / mut.rel_path
    if not target.exists():
        return False
    src = target.read_text()
    if mut.old not in src:
        return False          # stale pattern: refresh this row
    target.write_text(src.replace(mut.old, mut.new, 1))
    return True


def check(repo: Path, pin_cmd: list[str]) -> dict:
    baseline_exit = run(pin_cmd, repo)          # pins must be green on pristine tree
    caught = applied = 0
    detail: list[dict] = []

    for mut in MUTATIONS:
        with tempfile.TemporaryDirectory() as tmp:
            work = Path(tmp) / "repo"
            shutil.copytree(repo, work, ignore=IGNORE)
            if not apply_mutation(work, mut):
                detail.append({"mutation": mut.name, "status": "stale-pattern"})
                continue
            applied += 1
            hit = run(pin_cmd, work) != 0
            caught += int(hit)
            detail.append({"mutation": mut.name,
                           "status": "caught" if hit else "missed"})

    score = caught / applied if applied else 0.0
    return {"baseline_exit": baseline_exit, "applied": applied, "caught": caught,
            "score": round(score, 2), "detail": detail}


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--repo", default=".")
    ap.add_argument("--pin-cmd", required=True)
    ap.add_argument("--min-score", type=float, default=0.6)
    args = ap.parse_args()

    report = check(Path(args.repo).resolve(), args.pin_cmd.split())
    print(json.dumps(report, indent=2))

    if report["baseline_exit"] != 0:
        print("harness error: pins already fail on the pristine tree", file=sys.stderr)
        return 2
    return 0 if report["score"] >= args.min_score else 1


if __name__ == "__main__":
    raise SystemExit(main())

The mutation list is the actual deliverable. Each row is a belief you are testing. Add a row every time a reviewer asks "but does this break X?"

Step 1: Confirm the pristine tree is green

Run the pins alone first. A failing baseline invalidates the whole measurement.

python -m pytest -q tests/pins    # must exit 0
python pin_check.py --repo . --min-score 0.6 \
  --pin-cmd "python -m pytest -q tests/pins"

Expect baseline_exit: 0. Expect a detail array. Expect missed rows. The misses are the point.

Step 2: Read the score as a gate, not a grade

The number is a threshold check. Use it to decide whether to start, not to celebrate.

A high score is not automatically good news. Suites that pin every internal call also score well. They just cost you every future rename.

Step 3: Probe for tautological pins

Pins that import private names rarely specify behavior. They duplicate it.

# Suspects: pins reaching into underscore-prefixed helpers.
grep -rnE "^\s*from [A-Za-z_.]+ import _" tests/pins | tee tautology-suspects.txt
wc -l < tautology-suspects.txt

Review each hit. Ask one question: does this assertion describe output, or restate the source? If it restates, rewrite it against the public boundary.

Step 4: Make the smallest safe change

One behavior-preserving step. One commit. No drive-by cleanups, no renames.

git switch -c refactor/invoice-totals
# single change here only
python -m pytest -q tests/pins      # green after the change
git stash
python -m pytest -q tests/pins      # green before it too
git stash pop

If the pins pass only on the changed tree, they were tailored to the change. That is a rewrite of the specification, not a verification of it.

Where free tooling fits

The mutation list is the slow part. Someone has to guess which behaviors matter. I use MonkeyCode's free model access to draft candidate rows from a module's recent diff history, then delete most of them by hand. The candidates are proposals only, and every row still needs a human to confirm it describes real behavior. Its free server option is useful when the pin suite is too heavy for a laptop and CI minutes are budgeted elsewhere.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat availability, limits, and model lineups as operator-supplied and changeable. Verify current terms yourself. Do not put a free tier on a release-critical path.

Limitations and who should skip this

Text-level mutations are crude. They catch only a pattern that still exists verbatim. Rename a variable and the row silently becomes stale-pattern. Refresh rows when you touch the file.

A high score does not prove correctness. It proves the net can fail on the mutations you thought of. Semantic drift outside your list stays invisible.

If you already run a real mutation framework, use it. Tools like mutmut or cosmic-ray generate far better mutants than a hand-written list. This harness is for repos where adding one more test dependency is politically harder than adding one script.

Skip this if the output has no stable entry point, or if nobody consumes the output. Skip it for throwaway scripts. The score only matters where a regression has a cost.

Start with the misses

Run the harness on your messiest module and post the missed list. The score is a gate. The misses are the map.

One behavior-preserving change, one verified net. In that order.