判斷 Agent 程式碼修改前，先凍結不穩定測試

A reviewer opens an agent patch at 2am. CI is red. The diff touches a cache layer and a date formatter.

Nothing in the diff explains the failure. So the reviewer reruns the job. It goes green.

That single rerun is the problem. An agent patch has no author to ask. Its only defense is the verdict, and a flaky suite makes every verdict negotiable.

This is a plan for three layers: property checks, pinned fixtures, and a hard freeze on flaky tests. The freeze is the part most teams skip, and it is the part that decides whether the other two matter.

Why the freeze comes first

Think of a flaky test as a smoke alarm wired to a toaster. You cannot remove it, because it might be real. You cannot trust it, because it fires on nothing.

Every rerun is a human deciding which alarm to ignore. That decision is invisible in the log. Two reviewers can reach opposite verdicts on the same patch and both look reasonable.

Agents make this worse in a specific way. A model that sees a red run will often patch around the failure, not the cause. You get a green build and a diff that now depends on ordering luck.

So before the agent runs, the suite needs a declared set of tests whose signal is known to be unreliable. Not deleted. Not quarantined silently. Frozen, dated, and owned.

Step one: measure flakiness, do not guess

Run each suspect test many times in isolation. Record the pass rate. Anything between all-pass and all-fail is flaky by definition.

# tools/flake_sweep.py
import collections
import subprocess
import sys

RUNS = int(sys.argv[1]) if len(sys.argv) > 1 else 7
NODES = [l.strip() for l in open("tests/suspects.txt") if l.strip()]

results = collections.defaultdict(list)

for node in NODES:
    for _ in range(RUNS):
        code = subprocess.run(
            [sys.executable, "-m", "pytest", node, "-q", "-p", "no:cacheprovider"],
            capture_output=True,
        ).returncode
        results[node].append("pass" if code == 0 else "fail")

for node, outcomes in sorted(results.items()):
    rate = outcomes.count("pass") / len(outcomes)
    verdict = "STABLE" if rate in (0.0, 1.0) else "FLAKY"
    print(f"{verdict:6} {rate:4.2f} {RUNS:3} {node}")

Seven runs is a starting budget, not a proof. A test with a 90% pass rate survives seven runs about half the time. Raise the run count for anything near the boundary and treat the number as a lower bound.

Isolation matters more than volume. A single-process sweep with shared state will tag honest tests as flaky and hide the real offenders.

Step two: make the freeze expire

A freeze with no deadline becomes a permanent mute button. Write it with an owner and a review date, then let the gate fail when the date passes.

# tests/flaky-freeze.toml
[freeze]
owner = "platform"
review_by = "2026-11-30"
reason = "ordering-dependent cache eviction; ticket PLAT-4412"

# tools/freeze_gate.py
import datetime
import pathlib
import sys
import tomllib  # Python 3.11+

path = pathlib.Path("tests/flaky-freeze.toml")
if not path.exists():
    sys.exit("no freeze manifest; run the sweep first")

manifest = tomllib.loads(path.read_text())
review_by = datetime.date.fromisoformat(manifest["freeze"]["review_by"])

if datetime.date.today() > review_by:
    sys.exit(f"flaky freeze expired on {review_by}: re-sweep or delete entries")
print("freeze valid")

The expiry turns an excuse into a task. The owner sees the date in the manifest long before the gate turns red.

Step three: property checks before examples

Agents break invariants far more often than they break specific examples. A test named test_refund_1250 tells you almost nothing about a rewritten refund path.

A property check states the rule that must hold for every input. Balance never goes negative. Ordering never duplicates an event id. Serialization round-trips.

# tests/test_ledger_properties.py
from hypothesis import given, settings, strategies as st

@given(st.lists(st.integers(min_value=-10**6, max_value=10**6), max_size=200))
@settings(max_examples=200, deadline=None, derandomize=True)
def test_ledger_total_is_order_independent(entries):
    forward = sum(entries)
    backward = sum(reversed(entries))
    assert forward == backward

derandomize=True keeps the failure reproducible. Without it, a red property run is one more thing nobody can reproduce.

Drafting property candidates from a diff is tedious and models are reasonable at it. That is where I lean on MonkeyCode's free model access: I paste the changed function and ask for invariants, then keep only the ones I can state in one sentence.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step four: pin the environment, not just the data

Fixtures drift in three places at once: records, clock, and hash seed. Pin all three or the sweep measures your machine, not the code.

PYTHONHASHSEED=0 pytest -q --tb=short -k "not frozen"

The sweep is the only CPU-heavy row. It is also the only row where a bad run costs nothing but time.

Running it on a laptop means it gets skipped. The free server option is a better home for a nightly sweep, and it keeps the freeze manifest honest without a dedicated runner.

What this plan does not fix

A freeze hides real failures in the frozen tests. That is the trade, and it is why the expiry date is not optional. If a frozen test guards a payment path, freeze with a failing assertion replaced by a named ticket, not a silent skip.

Small samples cannot separate flakiness from a real bug. If a test fails once in ten runs, you have one data point and a suspicion. Escalate to more runs before you freeze it.

Property checks need decent generators. A generator that only emits zeros will pass forever and prove nothing. Inspect the inputs, not just the assertions.

And this whole approach costs CI minutes. If your suite runs in under a minute and has never flaked, you do not need it. Adding a sweep to a deterministic suite is ceremony, not safety.

Who should skip it

Skip this if your test suite is small and deterministic. Skip it if your CI budget is already tight and cannot afford a nightly job. Skip it if there is no one willing to own the freeze manifest, because an unowned freeze is just an ignored test with extra steps.

If you do adopt one piece, adopt the manifest and its expiry date. Measured flakiness, with a deadline attached, is the cheapest thing on this list. You can run the first sweep tonight, and the free server option is the least disruptive place to try it.