為特徵測試建立 Kill Rate Gate，量度測試能否察覺行為變化

A green characterization suite proves very little. It proves the tests ran once against current code.

It does not prove the tests would fail if behavior changed. That is a separate property, and you can measure it.

The measurement is a kill rate. Break one function on purpose. Check whether the suite notices.

Get that number before you touch a messy module. This post builds the harness.

Why a Green Suite Can Be Vacuous

Most characterization tests are written fast. You snapshot a return value, or assert result is not None.

Neither assertion constrains behavior much. Almost any edit still passes them.

The failure mode is quiet. The suite stays green through a real regression, and you trust it anyway.

A kill rate exposes this in minutes. It is a property of the suite, not of the code.

The Harness in Four Steps

Step 0 — Prove the baseline is stable

Run the suite twice against untouched code. Both runs must pass.

If either run fails, stop. You have flakes, not a safety net.

python -m pytest -q --tb=no; echo "exit=$?"
python -m pytest -q --tb=no; echo "exit=$?"

Flaky baselines poison the whole gate. Every mutation run inherits the same noise.

Step 1 — Enumerate functions and mutation operators

Save this as sabotage_gate.py. It needs Python 3.9 or newer, because it uses ast.unparse (stdlib docs).

# sabotage_gate.py — Python 3.9+ (needs ast.unparse). Run from the repo root.
import ast, collections, json, shutil, subprocess, sys, tempfile
from pathlib import Path

SKIP = (".git", ".venv", "venv", "node_modules", "__pycache__", "build", "dist")

class FlipCompare(ast.NodeTransformer):
    FLIP = {ast.Lt: ast.LtE, ast.LtE: ast.Lt, ast.Gt: ast.GtE, ast.GtE: ast.Gt,
            ast.Eq: ast.NotEq, ast.NotEq: ast.Eq, ast.In: ast.NotIn, ast.NotIn: ast.In}
    def visit_Compare(self, node):
        self.generic_visit(node)
        node.ops = [self.FLIP.get(type(op), type(op))() for op in node.ops]
        return node

class FlipBool(ast.NodeTransformer):
    def visit_BoolOp(self, node):
        self.generic_visit(node)
        node.op = ast.Or() if isinstance(node.op, ast.And) else ast.And()
        return node

class BumpInt(ast.NodeTransformer):
    def visit_Constant(self, node):
        if isinstance(node.value, int) and not isinstance(node.value, bool):
            return ast.Constant(value=node.value + 1)
        return node

OPERATORS = {"flip_compare": FlipCompare, "flip_bool": FlipBool, "bump_int": BumpInt}

Three operators keep the cost bounded. Flip comparisons, swap and/or, bump integer constants.

All are cheap to apply. Most are semantics-changing, which is exactly what you want.

Next, walk the file and collect qualified function names.

def functions(tree):
    found = []
    def walk(node, prefix):
        for child in ast.iter_child_nodes(node):
            if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)):
                found.append((prefix + child.name, child))
                walk(child, prefix + child.name + ".")
            elif isinstance(child, ast.ClassDef):
                walk(child, prefix + child.name + ".")
    walk(tree, "")
    return found

Step 2 — Mutate one function at a time in a scratch copy

The script copies the repo into a temp directory. It rewrites exactly one file, then runs the suite there.

Restoring the original bytes after each run matters. ast.unparse drops comments and reformats everything.

That normalization is fine for a throwaway copy. It is not fine for anything you commit.

def mutate(src, qualname, op):
    tree = ast.parse(src)
    for name, node in functions(tree):
        if name == qualname:
            t = OPERATORS[op]()
            node.body = [t.visit(stmt) for stmt in node.body]
            ast.fix_missing_locations(tree)
            return ast.unparse(tree)
    raise LookupError(qualname)

def run(cmd, cwd, timeout):
    try:
        p = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=timeout)
        return p.returncode, (p.stdout + p.stderr)[-400:]
    except subprocess.TimeoutExpired:
        return 124, "<timeout>"

A timeout is not a pass. Treat exit code 124 as a separate verdict, not as a survival.

Step 3 — Run the loop and collect verdicts

The main loop enforces a green baseline twice, then iterates. One mutation, one suite run, one restore.

def main(src_root, test_cmd, timeout=120.0, per_module=40):
    root = Path.cwd()
    work = Path(tempfile.mkdtemp(prefix="sabotage-")) / "repo"
    shutil.copytree(root, work, ignore=shutil.ignore_patterns(*SKIP))

    for _ in range(2):                        # baseline must be green twice
        code, tail = run(test_cmd, work, timeout * 5)
        if code != 0:
            sys.exit("baseline not green — fix flakes first:\n" + tail)

    rows = []
    for path in sorted(Path(src_root).rglob("*.py")):
        if set(path.parts) & set(SKIP):
            continue
        rel, src = path.relative_to(root), path.read_text()
        for qualname, _ in functions(ast.parse(src))[:per_module]:
            for op in OPERATORS:
                (work / rel).write_text(mutate(src, qualname, op))
                code, tail = run(test_cmd, work, timeout)
                (work / rel).write_text(src)   # restore original bytes
                rows.append({"module": str(rel), "target": qualname, "op": op,
                             "killed": code != 0, "timed_out": code == 124, "tail": tail})

    Path("mutations.json").write_text(json.dumps(rows, indent=2))
    agg = collections.defaultdict(lambda: [0, 0])
    for r in rows:
        agg[r["module"]][0] += r["killed"]
        agg[r["module"]][1] += 1
    for mod, (k, n) in sorted(agg.items()):
        print(f"{k/n:5.0%}  {k:3d}/{n:<3d}  {mod}")

if __name__ == "__main__":
    root = sys.argv[1] if len(sys.argv) > 1 else "src"
    cmd = sys.argv[2:] or [sys.executable, "-m", "pytest", "-x", "-q", "--tb=no"]
    main(root, cmd)

Run it with the source root, then your test command.

python sabotage_gate.py src python -m pytest -x -q --tb=no

Budget the runtime up front. Forty functions × three operators × four seconds is roughly eight minutes.

Lower per_module when that is too slow. Sample deterministic functions first, not I/O-heavy ones.

Step 4 — Gate on the modules you are about to touch

The summary line per module is the decision input. A workable default threshold is 0.6 on the blast radius.

Below that, shrink the refactor. Fewer functions per commit, or more tests first.

Keep mutations.json. The survivor entries are your test-writing backlog, ranked by real risk.

Reading Survivors, Not Just the Score

A survivor has two possible meanings, and they are not equivalent.

An equivalent mutant is behaviorally identical to the original. Example: bumping an integer inside a bound that is recomputed on every call.

Untested behavior is a live branch nobody asserts on. That one is your problem.

Manual triage slows down fast, usually somewhere past the hundredth mutation. This is a reasonable place for a free model to help classify: send the function body plus the surviving mutant, and ask for an input that distinguishes them.

Treat the answer as a hypothesis. Run that input in a REPL before you believe it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The loop is CPU-bound and disposable, which fits the free server option well. A scratch machine runs mutations and throws the workspace away. Keep production credentials off it.

If the free option is not available to you, the script runs fine on any box with the repo's dependencies installed.

Decision Table

Limitations

• Equivalent mutants inflate the survivor count. Subtract them before drawing conclusions.
• ast.unparse normalizes formatting and drops comments. Never commit mutated files.
• Cost scales with mutations times suite runtime. Long suites need sampling, which weakens coverage claims.
• A kill can be a false positive. A mutation may break an import and fail an unrelated test. Check tail before trusting a kill.
• Kill rate is measured at one commit. It decays as code and tests drift apart.
• This measures detection, not correctness. It cannot tell you the expected behavior is right.

Who Should Not Use This

Skip it if you have no suite at all. Write ten characterization tests first; then come back.

Skip it on a repo with a hard five-minute CI budget unless you sample aggressively. The gate is minutes, not seconds.

Skip it if you need coverage percentages for compliance. This answers a different question, and mixing the two numbers confuses reviewers.

Do not run it against code you cannot restore. That means a clean working tree and a scratch copy, always.

A Practical Starting Point

Pick the module you are most afraid to touch. Write tests until the baseline is green twice, then run the loop.

The first kill rate will probably disappoint you. That number is the real starting point for the refactor, not the green checkmark.

Smallest safe change means smallest change your tests can actually detect. Now you can measure whether they can.