Agent 生成補丁測試重跑至通過並非證據，應為偶發失敗設定預算

Rerunning Agent Patch Tests Until Green Is Not Evidence — Budget the Flakes Instead

A green rerun is a coin flip with better marketing. If your merge gate for an agent-generated patch is "run the suite, retry until it passes," you are not measuring the patch — you are measuring how many attempts your CI budget allows.

The fix is not "more tests." It is three tiers with three different jobs: property checks that survive a rewrite, fixtures pinned to a digest, and a repeat budget that turns a suspected flake into an estimated rate with an upper bound. A quarantine entry is the only escape hatch, and it expires.

Everything below runs locally with pytest and the standard library. Total setup: one extra file, no services.

The arithmetic that kills rerun-to-green

Suppose a candidate patch introduces a real defect, and one test detects it on 15% of runs (a plausible number for anything touching timing, caches, or model output). An "any pass wins" policy with k attempts misses the bug with probability (1 - 0.15)^k.

This is arithmetic, not a benchmark I ran. The point stands anyway: retries convert a detectable defect into an undetectable one, and the conversion rate is set by the flakiest test in the selected set, not by the patch.

Three tiers, three different jobs

Each tier answers a different question, and mixing them is why gates get unreliable.

• Tier 0 — properties. Does the patch violate an invariant? Deterministic, cheap, runs once.
• Tier 1 — pinned fixtures. Did observable output change? Deterministic, runs once, fails on any diff.
• Tier 2 — repeat budget. Is this test stable enough to be treated as evidence at all? Non-deterministic, runs n times.

Only tier 2 may be quarantined. Tiers 0 and 1 have no flake concept: a property failure or a fixture diff is a signal, full stop.

Step 1 — Classify the patch before you run anything

Start with the shape of the diff, not the test names.

git diff --stat origin/main...HEAD
git diff --name-only origin/main...HEAD | grep -E '\.(py|ts|go)$' | xargs -r git diff -U0 -- | grep -E '^\+' | grep -cE 'await|thread|lock|time\.|random|retry'

That second count is crude but useful: concurrent or time-dependent lines pull the patch into a stricter row of the table below.

The repeat counts are not vibes. They come from the rule of three: with zero failures in n runs, the 95% upper bound on the true failure rate is about 3/n. n=30 gets you ≤10%, n=60 ≤5%, n=200 ≤1.5%.

Step 2 — Property checks that survive a rewrite

Agent patches often rewrite a function's interior while the intent stays the same. A property test asserts the intent.

# tests/test_properties.py
from hypothesis import given, settings, strategies as st
from app.normalize import normalize_event

ALLOWED_KEYS = {"id", "kind", "at", "payload"}

given_event = st.fixed_dictionaries({
    "id": st.text(min_size=1, max_size=64),
    "kind": st.sampled_from(["click", "view", "error"]),
    "at": st.integers(min_value=0, max_value=2**31),
    "payload": st.dictionaries(st.text(max_size=8), st.integers()),
})

@settings(max_examples=300, deadline=None)
@given(given_event)
def test_idempotent(event):
    assert normalize_event(normalize_event(event)) == normalize_event(event)

@settings(max_examples=100, deadline=None)
@given(given_event)
def test_schema_is_closed(event):
    assert set(normalize_event(event)) <= ALLOWED_KEYS

@settings(max_examples=100, deadline=None)
@given(given_event)
def test_ordering_independent(event):
    once = normalize_event(event)
    twice = normalize_event(dict(reversed(list(event.items()))))
    assert once == twice

Run these with PYTHONHASHSEED=0 pytest tests/test_properties.py -q. If a patch can only pass by weakening a property, that is the review conversation — not a retry.

Step 3 — Pin fixtures to a digest

Fixture diffs are the cheapest way to catch a silent behavior change, provided the pin is part of the diff.

# regenerate, then inspect the diff before committing the pin
pytest tests/test_contract.py -q --update-pins
# expected: tests/pins/ingest_contract.json changes AND the code diff explains why

# tests/pinned.py
import hashlib, json, pathlib

PIN_DIR = pathlib.Path("tests/pins")

def pin(name: str, payload: dict) -> None:
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256(blob.encode()).hexdigest()
    path = PIN_DIR / f"{name}.json"
    expected = json.loads(path.read_text())["digest"]
    assert digest == expected, (
        f"{name} changed: {digest[:12]} != {expected[:12]}."
        " Update the pin only if this diff is the intended behavior change."
    )

Example command: pytest tests/test_contract.py -q. Exit code 1 plus a digest mismatch is a signal, not a flake. Never quarantine tier 1.

Step 4 — Measure the flake rate, do not guess it

This is the only file you need to add for tier 2.

# flake_budget.py
import json, os, subprocess, sys

BUDGET = {"max_upper_rate": 0.02, "min_runs": 200, "max_quarantined_share": 0.05}
LEDGER = "flake_ledger.json"


def wilson_upper(fails: int, n: int, z: float = 1.96) -> float:
    """95% upper bound for a binomial proportion."""
    if n == 0:
        return 1.0
    p = fails / n
    denom = 1 + z * z / n
    center = p + z * z / (2 * n)
    margin = z * ((p * (1 - p) / n + z * z / (4 * n * n)) ** 0.5)
    return min(1.0, (center + margin) / denom)


def run_once(nodeid: str, seed: int) -> bool:
    env = {**os.environ, "PATCH_SEED": str(seed), "PYTHONHASHSEED": "0"}
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", nodeid, "-q", "--no-header", "-p", "no:cacheprovider"],
        capture_output=True, text=True, env=env,
    )
    return proc.returncode == 0


def measure(nodeid: str, n: int, base_seed: int = 9137) -> dict:
    fails = sum(0 if run_once(nodeid, base_seed + i) else 1 for i in range(n))
    return {
        "nodeid": nodeid, "n": n, "fails": fails,
        "upper95": round(wilson_upper(fails, n), 4),
        "seed_base": base_seed,
    }


def load_ledger() -> dict:
    try:
        return json.loads(open(LEDGER).read())
    except FileNotFoundError:
        return {}


def gate(results: list, ledger: dict, today: str) -> int:
    exit_code = 0
    for r in results:
        entry = ledger.get(r["nodeid"])
        expired = entry is not None and entry["expires"] < today
        within_budget = r["upper95"] <= BUDGET["max_upper_rate"] and r["n"] >= BUDGET["min_runs"]
        if expired:
            print(f"FAIL {r['nodeid']}: quarantine expired on {entry['expires']}")
            exit_code = 1
        elif within_budget:
            print(f"OK   {r['nodeid']}: upper95={r['upper95']} over {r['n']} runs")
        elif entry is None:
            print(f"FAIL {r['nodeid']}: upper95={r['upper95']} > {BUDGET['max_upper_rate']}, no quarantine entry")
            exit_code = 1
        else:
            print(f"WARN {r['nodeid']}: quarantined until {entry['expires']} (owner {entry['owner']})")
    return exit_code


if __name__ == "__main__":
    targets = [line.strip() for line in open("flake_targets.txt") if line.strip()]
    runs = int(os.environ.get("FLAKE_RUNS", BUDGET["min_runs"]))
    results = [measure(t, runs) for t in targets]
    json.dump(results, open("flake-report.json", "w"), indent=2)
    sys.exit(gate(results, load_ledger(), os.environ["TODAY"]))

Run it:

printf 'tests/test_ingest.py::test_stream_backpressure\n' > flake_targets.txt
FLAKE_RUNS=200 TODAY=2026-10-15 python flake_budget.py; echo "exit=$?"

A 2-second test repeated 200 times is about 7 minutes of wall clock. That is the real cost of treating a timing-sensitive test as evidence instead of a rumor.

Step 5 — The quarantine ledger expires, always

A quarantine entry is a debt with a due date and an owner.

{
  "tests/test_ingest.py::test_stream_backpressure": {
    "class": "flake",
    "seed_base": 9137,
    "n": 200,
    "fails": 5,
    "upper95": 0.0537,
    "owner": "@finley",
    "expires": "2026-11-14",
    "evidence": "ci/run-88213"
  }
}

Three rules keep the ledger from becoming a graveyard:

1. Expiry is mandatory and enforced in code, not in the PR description.
2. The quarantined share of the selected test set is capped (5% here); crossing the cap fails the gate.
3. Tier 0 and tier 1 node IDs are rejected if they appear in the ledger at all.

Where a free-tier runner actually helps

The step-4 harness is CPU-bound and boring: 200 serial repeats per target, run on every candidate patch. That is exactly the workload that fits a free server option — you do not need a persistent runner for a nightly flake census. MonkeyCode offers free model access and a free server option as of this writing, which is how I would generate patch candidates and then run the repeat matrix without provisioning a box.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest caveats. I have not benchmarked its models against anything, and free-tier availability and limits change — read the current terms before you build a gate on top of it. Also keep the harness runner-independent: the flake_budget.py above shells out to pytest and does not import any vendor SDK, so swapping the runner is a one-line change.

If you want to try the matrix without provisioning first, the free server option is a reasonable place to start; the reproduction steps above are the part that matters.

Limitations and who should not use this

• Statistical gates are not proofs. upper95 <= 0.02 over 200 runs means "we did not observe frequent failure," not "this test is deterministic." A 1-in-500 flake passes a 200-run budget.
• Serial repeats are slow. 200 runs × 5 flaky-prone targets × 2 seconds is ~33 minutes. If your CI budget cannot absorb that, cap tier 2 at 30 repeats and accept the weaker claim — do not silently skip the measurement.
• Skip this approach entirely if your suite is already deterministic and fast, if your environment forbids quarantine of any kind (regulated release processes), or if your tests mutate shared production-adjacent state — repeating them 200 times multiplies the blast radius, not the signal.
• Do not apply it to tier 0 or tier 1. Quarantining a property check converts a defect into a policy.

Copy-paste checklist

1. Classify the patch with the diff-count command and pick the tier-2 row from the decision table.
2. Add or extend tier-0 property checks; confirm they fail on a deliberately broken candidate before you trust them.
3. Verify tier-1 pins changed only where the code diff explains the change.
4. Run FLAKE_RUNS=<n> TODAY=<date> python flake_budget.py and archive flake-report.json next to the patch.
5. For every non-compliant target, write a ledger entry with an owner and an expiry — or fix the test.
6. Re-run the census after the expiry date; an expired entry is a failed gate, not a warning.