Kill-Matrix 剪枝：剔除無法捕捉 AI C++ 補丁缺陷的黃金測試案例

A team running an AI patch pipeline once showed me their nightly eval: 214 golden cases, a clean green dashboard, and a production regression that the suite had walked straight past. The postmortem took an hour. Then someone asked the obvious question, and nobody had an answer: how many of those 214 cases had ever failed for a reason worth knowing?

We instrumented the suite instead of arguing about it. Sixty-one cases had never failed on any patch the pipeline had ever scored. Thirty-three failed on nearly every patch, which made them noise rather than signal. The suite looked thorough and was mostly ceremony.

This article describes the instrument we built: a kill matrix. It is a small, reproducible way to find out which golden cases actually earn their runtime, and which ones you are paying for out of habit. A free-tier model and a free-tier runner make the measurement cheap enough to run weekly, but the method stands on its own.

Step 1: Define the mutants before you look at the cases

A kill matrix needs a fixed oracle side. That side is a set of known-wrong C++ patches, called mutants, seeded by hand or by a mutation tool. The rule: every mutant must be a bug a reviewer would reject in a real PR.

Start with twenty. Small, boring, semantics-bearing mutations work best:

// original
std::size_t clamp_index(std::size_t i, std::size_t n) {
    return i < n ? i : n - 1;
}

// mutant M07: boundary flipped, one-past-end reachable
std::size_t clamp_index(std::size_t i, std::size_t n) {
    return i <= n ? i : n - 1;
}

// mutant M12: unsigned underflow when n == 0
std::size_t clamp_index(std::size_t i, std::size_t n) {
    return i < n ? i : n - 1;  // unchanged, but n == 0 now allowed by caller
}

Record each mutant as a patch file plus a one-line description of the defect. Do not include anything you cannot defend in a code review. Style-only mutants produce a matrix full of false kills.

Step 2: Build the matrix with a fixed seed and a fixed compiler

The matrix is a two-dimensional table: mutants down the rows, golden cases across the columns. Each cell answers one question — did this case fail when it saw this mutant?

The runner below is deliberately dumb. It compiles the mutant, applies the case, and writes a boolean. No model, no grader, no rubric. The point of this pass is to measure the cases, not the model.

# build_kill_matrix.py
import json, pathlib, subprocess, sys

ROOT = pathlib.Path(__file__).parent
MUTANTS = sorted((ROOT / "mutants").glob("*.patch"))
CASES = sorted((ROOT / "cases").glob("case_*.py"))

def run_case(case: pathlib.Path) -> bool:
    """Return True when the case FAILS (i.e. the mutant is killed)."""
    proc = subprocess.run(
        [sys.executable, str(case)],
        cwd=ROOT / "build",
        capture_output=True,
        timeout=120,
    )
    return proc.returncode != 0

matrix = {}
for mutant in MUTANTS:
    subprocess.run(["git", "apply", str(mutant)], cwd=ROOT, check=True)
    subprocess.run(["cmake", "--build", "build", "-j"], cwd=ROOT, check=True)
    row = {}
    for case in CASES:
        row[case.stem] = run_case(case)
    matrix[mutant.stem] = row
    subprocess.run(["git", "checkout", "--", "."], cwd=ROOT, check=True)

(ROOT / "kill_matrix.json").write_text(json.dumps(matrix, indent=2))
print(f"{len(MUTANTS)} mutants x {len(CASES)} cases written")

Two hygiene rules keep the matrix honest. First, freeze the compiler and flags for the whole run; a matrix built across two toolchains is two matrices stapled together. Second, run each cell once and record timing, because a case that takes four minutes to kill one mutant is a different purchase than one that takes four seconds.

Step 3: Read the matrix in four buckets

The raw JSON is not the deliverable. The classification is. Sort each case by how many mutants it kills, then look for the four shapes that matter.

The "never fires" bucket is where the silent regressions hide. A case with zero kills is either asserting something no realistic bug can violate, or it is silently skipping itself — a missing fixture, a disabled #ifdef, a test binary that swallowed a nonzero exit. Both need a human, not a deletion.

Step 4: Compress with greedy set cover, then validate on a holdout

Once you know which cases kill which mutants, the minimal suite is a set cover problem. Greedy selection gets you within a constant factor of optimal and takes about thirty lines.

# prune_suite.py
import json

matrix = json.load(open("kill_matrix.json"))
mutants = list(matrix)
cases = list(next(iter(matrix.values())))

killers = {c: {m for m in mutants if matrix[m][c]} for c in cases}
chosen, uncovered = [], set(mutants)

while uncovered:
    best = max(cases, key=lambda c: len(killers[c] & uncovered), default=None)
    if best is None or not (killers[best] & uncovered):
        break
    chosen.append(best)
    uncovered -= killers[best]

print(f"kept {len(chosen)}/{len(cases)} cases; uncovered mutants: {sorted(uncovered)}")
json.dump(chosen, open("suite.json", "w"), indent=2)

Split the mutant set 70/30 before you run this. Prune on the 70 percent, then check how much of the 30 percent the reduced suite still kills. A pruned suite that drops below roughly ninety percent holdout coverage was fit to the mutants, not to the code.

Step 5: Put the sweep on a free runner and measure the spend

This is where the harness meets infrastructure. The account's outreach brief points at two MonkeyCode options — free model access and a free server option — and both are relevant here for one reason: the kill matrix is a volume job. Twenty mutants across a hundred cases is two thousand executions, and the interesting question is not whether you can afford it once, but whether you can afford it every week.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free tier as an unverified budget until your own numbers exist. The brief cites a free token allowance; that figure is a marketing claim until a script counts what your sweep actually consumes. Log tokens per mutant, wall-clock per case, and failures per run, then compare against the allowance. If the ratio is comfortable, schedule the sweep weekly. If not, run it on the top twenty cases only and prune quarterly.

# one sweep, three numbers worth keeping
/usr/bin/time -f "wall=%e s" python build_kill_matrix.py 2>> sweep.log
grep -c '"true"' kill_matrix.json >> sweep.log
wc -l sweep.log

The same caution applies to the free server. Nothing in the brief says how long it stays up, how fast it builds, or whether it survives a -j build of your project. Measure those three things on day one, and keep a fallback runner configured so a capacity surprise costs you one sweep, not one quarter.

Step 6: Guard the pruning against its own success

Aggressive compression has a failure mode: the suite becomes fast, green, and blind. Three cheap guards prevent it.

1. Holdout mutants. Keep the thirty percent permanently out of selection, as in Step 4.
2. Canary cases. Add two cases that must fail on every run, one by design and one by an injected marker. If either passes, the harness is broken, not the code.
3. Quarterly re-seed. Models drift and so do bugs. Add ten fresh mutants every quarter drawn from real review rejections, and re-run the matrix.

Who should not do this

Skip this method if your mutant set would be invented rather than harvested from real rejections — a matrix of synthetic bugs measures your imagination, not your suite. Skip it if you have fewer than roughly fifteen cases, because set cover over a tiny universe just renames the problem. And skip it if the eval runs on a shared cluster where a two-thousand-execution sweep would starve other jobs; the measurement is not worth the queue.

What remains is a cheap habit. Once a quarter, you learn which of your golden cases still catch something, which never did, and what a full sweep actually costs on whatever free capacity you are leaning on. The dashboard stops being a number and starts being an argument.