沒有見證的屬性測試只是裝飾：智能體補丁的可證偽性閘門

A property that passes on the patched tree and on the pre-patch tree carries no information. That is the first thing to measure, before you trust any property an agent wrote for you.

Most agent-written property tests fail this way. The agent reads the diff, writes an invariant that the new code obviously satisfies, and the suite goes green in both trees. The test is not wrong. It is simply unable to distinguish the two states, which makes it a decoration, not a check.

This post describes a gate with three rules and one artifact. Properties need a committed witness and a measured detection rate on the baseline. Fixtures must be isolated per sample, verified by transcript hashing. Flaky-frozen tests are forbidden from acting as oracles. The gate is small enough to run on every agent patch.

The three roles, and what each one can lie about

Each row fails independently. A property with a real domain can still be useless if the fixture leaks state between samples; a clean fixture cannot save an invariant that holds by construction.

Step 1: Commit a witness next to every property

A witness is the concrete input that proves the property can fail. It is not a test case the agent is allowed to invent during the run — it lives in the repository, reviewable, in its own file.

{
  "id": "inv_total_never_negative",
  "seed": 20261018,
  "input": {"lines": [{"qty": 1, "unit_cents": -5}], "discount_cents": 0},
  "expected_on_baseline": "fail",
  "expected_on_patched": "pass"
}

The two expected_* fields are the whole point. On the merge-base tree the witness must reproduce the bug. On the patched tree it must pass. If the witness passes on both, the patch fixed nothing the witness can see, and the review should stop there.

# gate/witness.py — runnable sketch; plugin points are stubbed
import json, subprocess

def replay(tree_dir, prop_id):
    w = json.load(open(f"witnesses/{prop_id}.json"))
    return subprocess.run(
        ["python", "-m", "props.run", prop_id, json.dumps(w["input"])],
        cwd=tree_dir, capture_output=True, text=True,
    ).returncode == 0

Step 2: Measure detection power on the baseline, not on the patch

Passing the witness is necessary and cheap. The expensive question is whether the property generalizes past that one input. Sample the declared domain under a seeded generator on the pre-patch worktree, and count how often the invariant fails.

# gate/power.py
import random

def detection_power(prop, baseline_tree, gen, run_invariant, fixture_hash):
    rng = random.Random(prop.seed)
    fails, transcripts = 0, set()
    for _ in range(prop.samples):
        example, transcript = gen.sample(prop.domain, rng)
        transcripts.add(fixture_hash(transcript))       # rule 2, below
        if not run_invariant(baseline_tree, prop.invariant, example):
            fails += 1
    return fails / prop.samples, transcripts

Set power_floor from your own baseline run, not from this post. A property whose fail-rate on the buggy tree is near zero is a description of the current code, not a check on it. A fail-rate near one usually means the generator produces out-of-domain inputs and the property is failing for the wrong reason.

Rates in between need a human eye. There is no universal threshold; the number is a policy knob your repo owns and reviews.

Step 3: Isolate fixtures per sample and prove it with hashes

Property sampling multiplies whatever your fixture does. If setup writes rows, mutates a module-level cache, or advances a clock, sample 12 is running against sample 11's leftovers — and your fail-rate is measuring ordering, not behavior.

Give each sample a reset hook and hash the setup transcript (touched tables, env keys, temp paths, fake clock value). Across a run, the set of hashes must have exactly one member.

power, transcripts = detection_power(prop, baseline_tree, gen, run_invariant, fixture_hash)
if len(transcripts) != 1:
    fail(f"{prop.id}: fixture state varies across samples → {sorted(transcripts)[:3]}")
if power < prop.power_floor:
    fail(f"{prop.id}: detection power {power:.2f} < floor {prop.power_floor:.2f}")

A set of size greater than one is a deterministic, reproducible red — the kind of failure you can hand back to the agent without a discussion.

Step 4: Freeze the flaky tests, and keep them out of the oracle set

Freezing a flaky test so the suite stays green is a reasonable maintenance move. Letting that frozen test quietly become the only thing validating an agent patch is not.

Pass the freeze list into the property runner explicitly. Any intersection between the frozen set and the tests the property run actually executed is a hard failure, not a warning.

frozen = {l.strip() for l in open("tests/frozen.txt") if l.strip()}
ran    = run_property_suite(patched_tree, prop.id)      # returns test node ids
leaked = frozen & ran
if leaked:
    fail(f"{prop.id}: frozen tests used as oracle → {sorted(leaked)}")

The same list should carry an expiry date per entry. A freeze without an end date silently becomes a permanent hole in coverage.

Step 5: Wire it as one gate with three exit codes

Run the three steps in a fixed order and stop at the first failure. Exit code 0 means the property is falsifiable, its fixture is deterministic, and its oracle is un-frozen. Anything else is a reviewable artifact, not a vibe.

git worktree add ../baseline "$(git merge-base HEAD origin/main)"
python -m gate.run --prop inv_total_never_negative \
  --baseline ../baseline --patched . --samples 200 --power-floor 0.10
# 0 = accepted, 1 = property rejected, 2 = harness error
git worktree remove ../baseline

Per-property cost is samples × setup_time. Two hundred samples across a ten-property patch is a few minutes of wall clock on a modest machine — acceptable per pull request, not acceptable on every keystroke.

The proposal side of this loop does not need a frontier model; scanning a diff and drafting invariants is well inside what free model access handles, and running the sampler on the free server option keeps the 200-sample loop off a laptop that is already running the app under test. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability of free tiers is not a guarantee — check the current terms on your own account before wiring it into CI, and do not send proprietary code or credentials to a shared runner.

Limitations and who should skip this

• Property sampling only works on domains you can generate. Stateful protocols and payment flows usually need a hand-built model first.
• The power floor is a heuristic. It catches decorative properties; it does not prove the invariant is the right one.
• Suites with heavy fixtures (containers, seeded production snapshots) will spend the whole budget on setup resets.
• Teams with data-residency constraints should run the harness entirely in-house; the remote runner is a cost convenience, not a requirement.

Start with the witness registry. It is the cheapest file to commit and the fastest way to discover that half of your agent's properties pass on both trees. When the power floor turns out to be the number you have to lower, that lower number is the finding worth writing down.