隔離不穩定測試前，如何用 Seed Matrix 建立可重現證據

A red line in CI after an agent patch is not one event. It is at least three: the patch broke a real invariant, the property check itself is non-deterministic, or the test was already failing on the baseline commit. Freezing the test makes all three look identical in your logs, and the one you most need to catch — the real regression — disappears behind a quarantine label.

My rule now: no test gets frozen until it has a witness at the baseline commit, reproducible at a named seed. This post is the harness and the decision table I use to produce that witness. It is deliberately boring: standard library only, no framework, one JSON ledger per tree.

The failure that forced this

An agent patch replaced a hand-rolled de-duplication loop with a set. The diff looked like a cleanup. The order-preservation test went red, someone labelled it flaky because it had "gone red for no reason before," and it was quarantined for two weeks.

The test was not flaky. It was the only thing in the suite that noticed the ordering guarantee had been dropped. Nothing else failed, because nothing else asserted order.

If I had run that property across a handful of seeded inputs and stored the outcomes for both trees, the triage would have taken one command. Here is that command.

Step 1 — Make the seed an explicit input

The first mistake is letting the runner pick inputs. A property check with implicit randomness gives you a red line and no address for it. Seeds turn a failure into something you can cite.

# seed_matrix.py — stdlib only
import argparse, importlib, json, random, sys
from pathlib import Path


def load(spec):
    """spec = './trees/baseline::dedupe'"""
    dir_or_mod, func = spec.split("::")
    sys.path.insert(0, dir_or_mod)
    mod = importlib.import_module(Path(dir_or_mod).name)
    return getattr(mod, func)


def make_case(seed):
    rng = random.Random(seed)
    n = rng.randint(0, 12)
    return [rng.randint(0, 4) for _ in range(n)]


def order_preserved(impl, case):
    out = list(impl(case))
    expected = []
    for x in case:
        if x not in expected:
            expected.append(x)
    return out == expected


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--impl", required=True)      # tree::function
    ap.add_argument("--prop", required=True)      # module::function
    ap.add_argument("--seeds", type=int, default=200)
    ap.add_argument("--replays", type=int, default=5)
    ap.add_argument("--out", required=True)
    a = ap.parse_args()

    impl = load(a.impl)
    prop = load(a.prop)
    ledger = {"impl": a.impl, "prop": a.prop, "replays": a.replays, "seeds": {}}

    for seed in range(a.seeds):
        fails = 0
        for _ in range(a.replays):
            try:
                ok = prop(impl, make_case(seed))
            except Exception:
                ok = False
            fails += (not ok)
        ledger["seeds"][str(seed)] = fails

    Path(a.out).write_text(json.dumps(ledger, indent=1))


if __name__ == "__main__":
    main()

Run it once per tree, same property, same seed count:

python seed_matrix.py --impl ./trees/baseline::dedupe --prop seed_matrix::order_preserved \
  --seeds 200 --replays 5 --out baseline.json
python seed_matrix.py --impl ./trees/patched::dedupe --prop seed_matrix::order_preserved \
  --seeds 200 --replays 5 --out patched.json

One caveat that cost me an afternoon: replays are only independent if the implementation carries no process-level state. If it memoizes or caches, --replays 5 is five reads of the same answer. Run each replay in a subprocess, or reset the module between runs.

Step 2 — Triage with a decision table

The ledger stores fails out of replays per seed. That single number separates determinism from noise without any statistical machinery.

The last row matters as much as the first. A property that cannot fail is not a test, it is a comment with a runtime cost.

# triage.py — two ledgers in, one verdict out
import json, sys
from pathlib import Path


def load(p):
    d = json.loads(Path(p).read_text())
    return d["replays"], {int(k): v for k, v in d["seeds"].items()}


runs, base = load(sys.argv[1])
_, patch = load(sys.argv[2])
seeds = sorted(set(base) & set(patch))

det = lambda m, s: m[s] == runs
semi = lambda m, s: 0 < m[s] < runs

if all(patch[s] == 0 for s in seeds):
    print("NO_SIGNAL  green on the patch is not evidence")
elif any(det(patch, s) and base[s] == 0 for s in seeds):
    s = next(s for s in seeds if det(patch, s) and base[s] == 0)
    print(f"PATCH_BUG   witness seed={s} ({patch[s]}/{runs} patched, {base[s]}/{runs} baseline)")
elif any(semi(patch, s) for s in seeds):
    s = next(s for s in seeds if semi(patch, s))
    print(f"FLAKY_ORACLE   seed={s} {patch[s]}/{runs} on patch — not freezable yet")
elif any(patch[s] for s in seeds):
    s = next(s for s in seeds if patch[s])
    print(f"PRE_EXISTING   seed={s} {patch[s]}/{runs} patched and {base[s]}/{runs} baseline")
else:
    print("NEEDS_MANUAL_REVIEW")

Order matters. The PATCH_BUG branch runs before the flaky branch, because a deterministic failure on a tree that passes at baseline is a counterexample regardless of how noisy the rest of the matrix looks.

Step 3 — The freeze contract

Once triage says PRE_EXISTING or FLAKY_ORACLE, a quarantine entry is legitimate — but only with the witness attached. Mechanics of expiry are a separate discussion; the gate here is admission.

{
  "test": "tests/test_dedupe.py::test_order_preserved",
  "witness_seed": 137,
  "baseline_sha": "<commit that produced baseline.json>",
  "replay": "5/5 fail on baseline",
  "raised_by": "<your handle>",
  "review_by": "2026-11-11"
}

A quarantine entry that omits witness_seed or baseline_sha is not a freeze, it is a deletion with extra steps. In CI, reject entries missing either field. Keep the ledger file next to the entry — the JSON is the evidence, the entry is the pointer.

Where a remote free box actually helps

The matrix is the definition of a batch job: 200 seeds × 5 replays × 3 trees is 3,000 independent property evaluations with zero interactive latency requirement. That shape fits a free server instance well, and MonkeyCode's operator states that free model access and a free server option are available — I have not measured quotas, hardware, or runtime, so treat those as unverified for your workload.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two uses that hold up:

1. Property candidates from the diff. Ask a model to propose invariants that the changed function must preserve, then delete most of them. A candidate earns a place in the matrix only after you write a negative control — a known-bad input the property must reject. If the property accepts its own negative control, it cannot catch anything.
2. Running the matrix off your laptop. The baseline and patched ledgers need the same environment. A stock container gets you that, and keeps the 3,000 evaluations off the machine you are typing on.

Nothing in steps 1–3 requires either. If you already have a runner with stable seeds, skip the product entirely and keep the table.

Limitations, and who should not do this

• Seeds are not coverage. 200 seeds of a 12-element generator explore a tiny slice. The generator is the binding constraint, not the seed count. If your generator only produces small lists, the matrix proves small-list correctness and nothing else.
• Multiplies runtime. N seeds × K replays is N×K evaluations per property. On a suite that already takes 20 minutes, this is a targeted tool for one suspect property, not a policy for every test.
• Cannot fix wall-clock or network dependence. If the failure depends on a real clock, a real socket, or a shared database, seed control just makes the flakiness well-documented. You need fixtures.
• Needs subprocess isolation for stateful code. As noted above, in-process replays lie about independence for anything with caching.
• Skip it for small, fast, deterministic suites. If a property runs in milliseconds over a fully enumerated input space, the table is overhead.
• Do not run regulated or SLA-bound pipelines on free shared infrastructure. Run locally or on infrastructure you control.

If you use Hypothesis rather than the stdlib harness, the equivalent levers exist — the derandomize setting and the pytest plugin's seed flag — but flag names and behaviour have changed across versions. Check the installed version's docs before you quote the flags in a postmortem.

What to take away

A flaky label is a claim about the past, and claims about the past need a witness. Generate one seed, on the baseline tree, that fails 5 out of 5 times. Until you can name it, you are not looking at a flaky test. You are looking at an unexamined red line, and the cheapest explanation is usually the true one: the patch changed behaviour and only one test noticed.