Agent 程式碼修補的種子掃描審計：測試套件只在一種順序下通過

A green suite is a claim about one execution order.

Agent patches merge because the suite is green, and the suite runs the same way every time: file order, then declaration order, one seed. Coupling that the patch introduces between tests is invisible under that single order. Shuffle the order and the coupling shows up as a flip: same commit, same test, different verdict.

What follows is a four-step audit for that flip. Baseline a seed matrix, diff it across the patch, shrink each flip to a minimal triggering predecessor, and freeze only what reproduces — with an expiry date that CI enforces. The numbers in the examples are illustrative placeholders; your suite will produce different ones.

Three ways a patch creates order coupling

Most of it arrives through conftest.py, which agent patches edit freely because it looks like plumbing rather than product logic.

1. Fixture scope creep. A fixture moves from function to session scope to make the suite faster, and now the second test that needs it gets the residue of the first.
2. Shared mutable state. A module-level cache, temp path, database row, or environment variable is written by one test and read by another. Alone, each test passes.
3. Half-undone monkeypatching. A helper writes os.environ directly instead of using monkeypatch, so teardown never happens when an assertion fails earlier in the test.

None of these are exotic bugs. They are cheap edits a model makes when the failing test sits far away from the code that actually caused the failure.

Step 1 — Baseline the seed matrix before you merge

Install pytest-randomly and run your unit tier under a fixed seed list. Same seeds on main, same seeds on the patch branch, same machine class — timing-sensitive tests behave differently on slower runners.

pip install pytest-randomly
SEEDS="1 2 3 4 5 6 7 8" python scripts/order_matrix.py

# scripts/order_matrix.py
import json, os, subprocess, sys, xml.etree.ElementTree as ET

SEEDS = [int(s) for s in os.environ.get('SEEDS', '1 2 3 4').split()]
OUT = '.order-matrix'
os.makedirs(OUT, exist_ok=True)

def run(seed):
    xml = os.path.join(OUT, 'seed-{}.xml'.format(seed))
    subprocess.run(
        [sys.executable, '-m', 'pytest', '-q',
         '--randomly-seed={}'.format(seed), '--junitxml={}'.format(xml)],
        check=False,
    )
    results = {}
    for case in ET.parse(xml).getroot().iter('testcase'):
        key = '{}::{}'.format(case.get('classname'), case.get('name'))
        bad = case.find('failure') is not None or case.find('error') is not None
        results[key] = 'fail' if bad else 'pass'
    return results

matrix = {}
for seed in SEEDS:
    for test, verdict in run(seed).items():
        matrix.setdefault(test, {})[seed] = verdict
with open(os.path.join(OUT, 'matrix.json'), 'w') as fh:
    json.dump(matrix, fh, indent=2, sort_keys=True)

Run the script twice per commit. Two runs on the same commit tell you which tests are flaky regardless of order; a test that flips between seeds on the same commit is order-coupled, and that is the signal you are after.

Step 2 — Diff the matrices and rank by flip count

Flip count is simply the number of seeds where a test fails. Compare per test, not per suite, because totals hide swaps — one test starts passing for the wrong reason while another starts failing.

The last row is the one reviewers skip. A patch can make a flaky test look stable by adding a global reset or a cleanup fixture that masks the leak instead of removing it. Read the diff for deletion of coverage before you read it as a fix.

Step 3 — Shrink the flip to one predecessor

A failing seed tells you nothing actionable. You want the smallest pair: this test, preceded by that test, fails.

Record run order with a hook, guarded by an environment variable so it costs nothing in normal runs:

# conftest.py
import json, os

def pytest_runtest_logstart(nodeid, location):
    path = os.environ.get('ORDER_LOG')
    if path:
        with open(path, 'a') as fh:
            fh.write(json.dumps(nodeid) + '\n')

Then binary-search the prefix. The function below is an unexecuted template: run_prefix has to shell out to pytest with the given node IDs and return whether the target failed.

def first_trigger(run_prefix, order, lo=0, hi=None):
    # Smallest prefix whose execution makes the target fail.
    # Assumes leakage accumulates; verify the resulting pair by hand.
    hi = len(order) if hi is None else hi
    while lo < hi:
        mid = (lo + hi) // 2
        if run_prefix(order[:mid]):
            hi = mid
        else:
            lo = mid + 1
    return lo

Confirm with an explicit two-test command:

pytest tests/test_cache.py::test_reads_warm tests/test_api.py::test_patch -q

If the pair reproduces, you have a freeze record with a real command. If it does not, the leak is non-monotone — an earlier test masks it, or a later one triggers it — and you fall back to a pair sweep over the recorded order. That sweep is quadratic, which is why you only run it for tests that already flipped.

Step 4 — Freeze with an expiry that CI enforces

A skip is permanent. A freeze is a deadline. Record four fields per frozen test: the reproduction command, the seed list where it fails, an owner, and a date. Then make an expired freeze fail the build.

# conftest.py
import datetime, os, pytest

def pytest_collection_modifyitems(config, items):
    if os.environ.get('FREEZE_GATE') != 'enforce':
        return
    today = datetime.date.today()
    expired = []
    for item in items:
        for mark in item.iter_markers('flaky_freeze'):
            until = datetime.date.fromisoformat(str(mark.kwargs['freeze_until']))
            if until < today:
                expired.append('{} expired {}'.format(item.nodeid, until))
    if expired:
        pytest.exit('expired flaky freezes:\n' + '\n'.join(expired), returncode=1)

@pytest.mark.flaky_freeze(
    freeze_until='2026-11-08',
    owner='team-name',
    repro='pytest tests/test_cache.py::test_reads_warm tests/test_api.py::test_patch',
    issue='ENG-412',
)
def test_patch():
    ...

Keep the frozen test executing. The freeze means its failures do not block the merge, not that the assertion stops running — a silently skipped test cannot tell you when the upstream bug disappears. One line in a pytest_runtest_makereport hook, downgrading a failure on frozen tests to an xfail-style report, is enough.

Where a free tier actually fits this workflow

The sweep is embarrassingly parallel: seeds multiply jobs, and eight seeds on a twenty-minute suite is not something you want competing with your laptop. The runner itself is unremarkable — a single small box, the test dependencies, and a cron entry.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two concrete places it participates. First, the free server option can host the order-matrix runner for a small suite, which is the cheapest way to find out whether the audit is worth introducing at all. Second, the free model access is a reasonable fit for the judgement work that comes after the sweep: clustering flipping node IDs by their reproduction command and proposing likely candidates in conftest.py.

Treat those proposals as hypotheses only. Fixture edits change the environment for every test in the session, and the patch's own tests cannot verify that change — that is the whole reason the sweep existed. Start with eight seeds and one job before scaling anything, and keep the review of any fixture diff manual.

Limitations and who should skip this

• Runtime multiplies. Eight seeds on a twenty-minute suite is roughly two hours forty. Restrict the sweep to the unit tier, or use four seeds for slow suites and reserve eight for the tier around the changed files.
• Prefix bisection assumes accumulation. State leaks usually grow monotonically; nondeterministic races do not. Verify the pair before you write the freeze record.
• Some tests are legitimately ordered. Migration chains, sequence-number allocation, and integration fixtures that build a database stage by stage are not order-independent by design. Lock their scope explicitly instead of shuffling them.
• Timing flakiness will pollute the signal. If the baseline matrix already shows flips, fix or quarantine those first or you cannot attribute anything to the patch.
• Large monorepos pay a cache tax. Shuffled order defeats test-selection caches; run the sweep nightly rather than per pull request.

If nobody on the team owns the frozen tests, do not start. A freeze ledger without owners is a skip list with extra steps.

Merge checklist

1. Seed matrix recorded on main and on the patch branch, same seed list.
2. Every 0 → ≥2 flip either fixed or blocked.
3. Every ≥2 → 0 change explained in the pull request.
4. Each accepted freeze carries a reproduction command, an owner, and a date.
5. FREEZE_GATE=enforce runs on the default branch, not only locally.
6. The frozen test still executes and still reports.

The execution order is part of a patch's contract even when nobody wrote it down. Gate on it, or the next green run will be green for a reason you did not choose.