Agent 修補流程如何量度檢查閘門重疊，再決定是否增加第四道檢查

Most agent-patch pipelines fail by addition. A gate misses a bug, so you bolt on another gate, then another, until a 40-line patch triggers eleven minutes of checks nobody can attribute to a specific defect class. The redundancy stays invisible because each gate reports its own pass/fail and nothing reports the overlap.

Here is the working conclusion up front: before you add a fourth check, compute the Jaccard overlap between the three you already run. If two gates block the same probes, you are paying twice for one verdict, and the money you saved by merging them is the budget for a gate that actually covers a blind spot.

This article gives you a probe matrix, a small scoring script, and a freeze-accounting rule. Everything below is a template you run against your own repository; no numbers here are presented as measurements of a real suite.

The three gates worth stacking first

Agent patches break tests in three distinguishable ways, and each needs a different oracle.

These three are cheap to run and cheap to explain to a reviewer. That is exactly why people keep adding more of them, and exactly why the overlap problem appears.

Step 1 — Build a probe set, one class per probe

A gate's value is not how often it fails. It is how many distinct known-bad patches it catches. To measure that, you need known-bad patches, and each one needs exactly one label so a catch can be attributed.

Use two sources: reverted real fixes (revert the agent's fix, keep its tests — if the suite still passes, you have a tautology) and hand-written mutants confined to one class. Keep the labels coarse.

patch_id,probe_class,gate,verdict
p01,functional,property,fail
p01,functional,fixture,pass
p01,functional,flake,pass
p02,fixture,property,pass
p02,fixture,fixture,fail
p02,fixture,flake,pass

verdict=fail means the gate blocked that probe. probe_class=clean marks known-good patches that every gate must accept; those rows give you the false-block rate. Roughly 20 bad probes across the three classes is enough to see structure, and 40 is enough to trust the ordering. Write the CSV by hand the first time — it takes an afternoon and it is the only part of this you cannot automate away.

# gate naming must match the registry keys you use in CI
cut -d, -f3 probes.csv | tail -n +2 | sort -u
# fixture
# flake
# property

Step 2 — Run every gate against every probe

Do not run only the gate you believe in. The matrix is the point. Run all three gates on all probes, including the clean ones, and record the verdict even when you are sure it will pass.

Two practical notes. First, the flake gate needs repetition to be meaningful: a single run of a known-flaky test is a coin flip, not a verdict. Second, record the gate's wall-clock minutes in the same file or a sibling file, because cost is half of the decision and nobody remembers it a month later.

for g in property fixture flake; do
  /usr/bin/time -f "$g %e" ./run_gate.sh "$g" probes.csv
done 2>&1 | tee gate_minutes.txt

Step 3 — Score the matrix

The script below turns the matrix into four numbers per gate: catch rate, false-block rate, unique catches, and cost per unique catch. Unique catches are the only column that tells you whether a gate earns its place.

#!/usr/bin/env python3
"""gate_overlap.py — redundancy report for agent-patch gates.

usage: gate_overlap.py probes.csv property=4.5 fixture=1.0 flake=9.0
verdict: "fail" = gate blocked the patch, "pass" = gate accepted it.
probe_class == "clean" rows are known-good and must be accepted.
"""
from __future__ import annotations
import csv
import sys
from collections import defaultdict
from itertools import combinations


def load(path):
    with open(path, newline="", encoding="utf-8") as f:
        return list(csv.DictReader(f))


def build(rows):
    gates = sorted({r["gate"] for r in rows})
    blocked, false_block = defaultdict(set), defaultdict(set)
    bad, good = set(), set()
    for r in rows:
        pid, gate = r["patch_id"], r["gate"]
        verdict = r["verdict"].strip().lower()
        if verdict not in {"pass", "fail"}:
            raise SystemExit(f"bad verdict {verdict!r} at {pid}/{gate}")
        if r["probe_class"] == "clean":
            good.add(pid)
            if verdict == "fail":
                false_block[gate].add(pid)
        else:
            bad.add(pid)
            if verdict == "fail":
                blocked[gate].add(pid)
    return gates, blocked, false_block, bad, good


def jaccard(a, b):
    return 0.0 if not (a or b) else len(a & b) / len(a | b)


def main():
    if len(sys.argv) < 2:
        raise SystemExit("usage: gate_overlap.py probes.csv [gate=minutes ...]")
    rows = load(sys.argv[1])
    cost = {}
    for arg in sys.argv[2:]:
        gate, _, minutes = arg.partition("=")
        cost[gate] = float(minutes)

    gates, blocked, false_block, bad, good = build(rows)
    print(f"probes: {len(bad)} bad, {len(good)} clean\n")
    print("| gate | blocked | catch | false blocks | unique | min | min/unique |")
    print("|---|---|---|---|---|---|---|")

    for g in gates:
        others = set().union(*(blocked[h] for h in gates if h != g))
        unique = len(blocked[g] - others)
        catch = f"{len(blocked[g])}/{len(bad)}"
        fpr = f"{len(false_block[g])}/{len(good)}" if good else "0/0"
        m = cost.get(g)
        if m is None:
            per = "-"
        elif unique == 0:
            per = "never"
        else:
            per = f"{m / unique:.1f}"
        print(f"| {g} | {len(blocked[g])} | {catch} | {fpr} | {unique} | "
              f"{m if m is not None else '-'} | {per} |")

    print("\npairwise overlap (Jaccard of blocked sets):")
    for a, b in combinations(gates, 2):
        print(f"- {a} vs {b}: {jaccard(blocked[a], blocked[b]):.2f}")

    union = set().union(*blocked.values())
    missed = sorted(bad - union)
    print(f"\nblind spots (no gate blocked): {len(missed)}")
    for pid in missed:
        print(f"- {pid}")


if __name__ == "__main__":
    main()

Run it:

python3 gate_overlap.py probes.csv property=4.5 fixture=1.0 flake=9.0

Step 4 — Read the output as a decision, not a scoreboard

The report has one shape. The values below are a format example from a synthetic input — they are not a benchmark of any repository, including yours.

Read three things in this order.

1. Blind spots first. The blind spots list is the actual backlog. Gates are not the deliverable; coverage is. A gate with a great catch rate and two blind spots is worse than a mediocre gate with none.
2. Unique catches second. A gate with zero unique catches is not a gate, it is a duplicate with a longer runtime. It still has value as a defense in depth on the blocking path, but it should not be expanded.
3. Cost per unique catch last. Compare it against the cost of writing one more probe. If a gate costs nine minutes per unique catch and you can write three probes for the same effort, write the probes.

A useful heuristic when overlap is high: jaccard >= 0.6 with unique <= 1 means merge or demote. Demote means run it as a warning, not a blocker, so its failure does not consume the patch author's attention.

Step 5 — Count freezes honestly

A flaky test under freeze is a borrowed verdict. It cannot fail, which means every gate downstream of it is reading a pass that carries no information. If you do not account for it, your catch-rate table silently inflates.

Keep the registry machine-readable and require a counterexample, not a date, as the reason for the freeze:

# flake_registry.yaml
- test_id: tests/test_billing.py::test_proration_rounding
  frozen_at: 2026-10-24
  counterexample: 7e1c9a2   # commit where it failed twice on unchanged code
  owner: payments
  budget_runs: 200          # stop re-running after this; escalate instead
  expiry: must_fix_by_next_release

Then apply the accounting rule in the report: an active freeze contributes a pass to every gate, so subtract frozen tests from the denominator and print the count separately as coverage_unknown. A pipeline with 40 probes and 6 frozen tests has 34 real probes, not 40. Print both numbers and nobody can quietly widen the freeze list to make the table look better.

Enforce it in CI with a check that fails the build when a freeze has no counterexample or has outlived its stated condition:

python3 - <<'PY'
import yaml, sys
entries = yaml.safe_load(open("flake_registry.yaml")) or []
bad = [e for e in entries if not e.get("counterexample") or not e.get("owner")]
if bad:
    print("freeze without counterexample or owner:", [e["test_id"] for e in bad])
    sys.exit(1)
print(f"{len(entries)} active freezes accounted for")
PY

That is the whole freeze discipline: a flake is either reproduced with evidence and quarantined with an owner, or it is not frozen at all.

Where the free tier genuinely fits

The expensive part of this workflow is not the scoring. It is drafting probes and mutants, and re-running the flake gate many times to get a stable verdict. Both are bursty, low-stakes, and embarrassingly parallel.

That is where a free model tier plus a free server option is a reasonable fit: the model drafts candidate mutants and probe descriptions, and the harness runs the repeat-and-score loop off your paid CI runners. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am describing the workflow I would use, not reporting measured availability, throughput, or hardware — those claims are operator-supplied and change over time, so verify them yourself before relying on them.

Keep the boundary sharp. Drafting and dry-running belong on free capacity. The blocking decision belongs in your own CI, on infrastructure you control, because a gate that cannot be re-run on demand during an incident is not a gate. If the free tier queues or rate-limits under load, the probe-drafting step degrades gracefully; the merge decision must not.

Limitations and who should skip this

This approach measures the gates you already have. It cannot discover a defect class that none of your gates model, so a clean overlap report on a suite with no property tests just tells you your fixture lock is lonely.

It also assumes a probe set you can label. If reverting a fix rarely produces a still-green suite, your patches are too entangled for single-class probes, and you should fix that before measuring anything.

Skip this if you merge fewer than roughly twenty agent patches a week — the bookkeeping costs more than the redundancy it removes. Skip it if your suite is already too flaky to distinguish fail from flaky, because the flake column will be noise. And skip it if you cannot name an owner for a freeze entry; the registry will become a graveyard within a month.

If you do run it, run it once per quarter and diff the tables. Gate overlap is not a constant — it drifts as the suite and the patches change, and the quarter when fixture starts duplicating property is the quarter to merge them and spend the savings on the class you keep missing.