智能體程式碼補丁檢查如何按成本排序，並設計會到期的測試凍結機制

Most agent-patch gates I see are unordered. Every check runs on every patch, the expensive ones run first, and a twelve-minute suite is spent deciding that a diff has a whitespace conflict. The fix is not more checks. It is a ladder: order checks by cost, abort at the first blocking stage, and spend the remaining budget where the patch actually is.

This post is about sequencing and budget, not coverage. It assumes you already have some mix of static checks, targeted tests, property checks, and a full suite. The artifact here is a runner that enforces the order, and a freeze ledger that expires on its own terms instead of rotting.

The lever is order, not volume

A stage that costs 2 seconds and blocks 60% of bad patches should never sit behind a stage that costs 300 seconds and blocks 5%. That sounds obvious, and it is routinely violated in practice, because checks get added one at a time by whoever needed them last.

Measure two numbers per check type: median wall time, and the fraction of bad patches it blocks alone. The second number is the hard one. You get it from a small labeled set: ten to thirty patches you already know are bad, plus a handful you know are good. If you have no such set, build one before tuning the ladder — otherwise you are guessing at both axes.

With those two numbers, the ordering rule is simple: sort by cost / block_rate, ascending. Cheap-and-decisive first, expensive-and-fuzzy last.

A four-stage ladder that terminates early

Each stage is a hard gate. If s0 fails, nothing else runs, and the report says so. The runner records per-stage duration so the ordering can be re-derived from real data instead of intuition.

# ladder.py - cost-ordered stages with per-stage budget and hard abort.
# Dependency-free. Commands come from a JSON file so the runner stays repo-agnostic.
import json, subprocess, sys, time

STAGES = json.load(open(sys.argv[1]))  # [{"name":..., "budget_s":..., "cmd":...}, ...]

def run(cmd, budget):
    t0 = time.monotonic()
    p = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    elapsed = time.monotonic() - t0
    return {
        "code": p.returncode,
        "seconds": round(elapsed, 2),
        "over_budget": elapsed > budget,
        "stdout": p.stdout[-2000:],
        "stderr": p.stderr[-2000:],
    }

report, blocked_by = [], None
for stage in STAGES:
    r = run(stage["cmd"], stage["budget_s"])
    r["stage"] = stage["name"]
    report.append(r)
    if r["code"] != 0:
        blocked_by = stage["name"]
        break

print(json.dumps({"blocked_by": blocked_by, "stages": report}, indent=2))
sys.exit(0 if blocked_by is None else 1)

Run it as python ladder.py stages.json > ladder-report.json. The over_budget flag is advisory, not fatal: a stage that keeps blowing its budget is a signal to split it, not to delete it. Six weeks of those flags gives you a defensible cost table.

Steps to adopt it, in order:

1. Label 10–30 known-bad and 5 known-good patches. Do this first; everything downstream depends on it.
2. Assign each existing check to a stage by the cost / block_rate rule, not by which tool owns it.
3. Set budgets from observed medians, roughly 3× median, so a slow machine does not page anyone.
4. Run the ladder in report-only mode against your labeled set and confirm that stage 0 and 1 alone block the majority of bad patches.
5. Only then make stages blocking, cheapest first.

Properties belong at stage 2, generated not hand-written

Stage 2 is where agent patches usually escape, because agents edit code and tests together and the tests stay green. Property checks survive that pressure better than example tests, because a property states something the patched code cannot redefine: round-trip stability, idempotence of a normalizer, monotonicity of a ranking, conservation of item counts across a transform.

Enumerating those invariants by hand is slow. Drafting them from the diff with a model is a reasonable use of free model access — you paste the touched functions and ask for candidate invariants, then discard everything that restates the implementation. Keep only properties whose failure would be a product bug, not a formatting difference. The reference implementation above does not care where the properties came from; it only cares that they run inside a budget.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. In this account's workflow, MonkeyCode's free model access is a plausible host for that invariant-drafting step, and the free server option is a plausible host for the stage 2 and 3 harness when you do not want to provision a machine for a job that runs for minutes at a time. Both are availability claims from the operator, not benchmark results. Verify quotas, runtime limits, and data handling against current documentation before putting real patches through either.

The honest split: drafting invariants is a language task and fits free model access. Running the harness is a compute task and fits the free server. Neither replaces the labeled set from step 1, and neither tells you whether your properties are any good.

A freeze that re-arms itself

"Freeze the flaky test" is the standard advice and the standard way to lose a quarter of coverage. A freeze needs three fields to be safe: evidence, a fingerprint, and an expiry condition.

• Evidence: the test failed on the base commit without the patch. No evidence, no freeze. This is the difference between quarantining a real flake and muting a regression the patch introduced.
• Fingerprint: a hash of the test body. If the test is edited, the freeze is void immediately.
• Expiry: a fixed TTL, and a re-check on expiry. Whichever comes first, fingerprint change or TTL, ends the freeze.

# freeze.py - a freeze is void the moment the test body changes.
import hashlib, json, pathlib, sys, time

def fingerprint(path, name):
    src = pathlib.Path(path).read_text()
    body = src.split("def " + name + "(", 1)[-1]
    return hashlib.sha256(body.encode()).hexdigest()[:12]

def still_frozen(entry):
    if time.time() > entry["expires_at"]:
        return False, "ttl expired"
    if fingerprint(entry["file"], entry["test"]) != entry["fp"]:
        return False, "test body changed"
    return True, "held"

ledger = json.load(open(sys.argv[1]))
verdicts = {e["test"]: still_frozen(e) for e in ledger}
print(json.dumps(verdicts, indent=2))

Store the ledger next to the suite, review it weekly, and treat every "held" entry as debt with a date on it. A freeze you cannot re-check is a deletion with extra steps.

What this does not solve

This ladder does not tell you whether your properties are meaningful, and it cannot substitute for the labeled set — without known-bad patches, cost / block_rate is fiction. It also assumes checks are deterministic enough that a single failure means something; if base-vs-patch flake attribution is still guesswork in your repo, fix that before tuning the order.

Skip this approach if your suite finishes in under a minute (the ladder is overhead), if you have no labeled patches and no appetite to build a set, or if your patches are mostly generated in bulk and reviewed by a human who already reads the diff — the human is doing the stage 1 work and doing it better.

Adopt the ladder only up to the point where it changes a decision. For most teams that is stage 0 and stage 1, plus a freeze ledger that expires. Everything past that is tuning you should earn with data.

If you try this, publish the cost table you end up with. Per-repo numbers are the only ones that transfer.