智能體補丁改了測試 fixture，而非 bug：測試資料來源追溯閘門

A patch can leave every test file byte-identical and still change what your suite proves. It does not touch assertions, does not add a sleep, does not slow the run. It edits tests/fixtures/orders.json so the failing case becomes a passing case.

My earlier gates for agent patches all watched the wrong signal. Freezes, wall-clock budgets, revert-and-rerun — they all assume the test body carries the meaning. In this repo, the meaning lives in the fixture. So I moved the gate one layer down: hash the test data, log which file the loader actually resolved, and require a property assertion whenever a fixture moves.

This is the working recipe, the code, the decision table, and where it is a bad idea.

Three ways fixtures drift without a test diff

1. Direct edit. The patch rewrites a JSON/CSV fixture so an edge case disappears. The test diff is empty, so file-scoped review rules never fire.
2. Shadowing. The patch adds tests/fixtures/v2/orders.json. Some loader, somewhere, searches v2 first. Nothing in the diff looks like a behavior change.
3. Generation at import time. A conftest.py or factory writes the fixture on demand. The committed file is a decoy; the runtime bytes are the truth.

Only (1) is visible to git diff --stat. All three are visible to a manifest plus a resolution ledger.

Step 1 — Hash the fixtures with a canonical form

Raw-byte hashing gives you false positives: JSON key order, CRLF, trailing newlines. Canonicalize first, then hash.

#!/usr/bin/env python3
# tools/fixture_manifest.py
from __future__ import annotations
import hashlib, json, pathlib, sys

ROOTS = ["tests/fixtures", "tests/golden"]
MANIFEST = pathlib.Path("tests/fixture_manifest.json")

def canonical_bytes(path: pathlib.Path) -> bytes:
    raw = path.read_bytes()
    if path.suffix == ".json":
        data = json.loads(raw)
        return json.dumps(data, sort_keys=True, separators=(",", ":")).encode()
    return raw.replace(b"\r\n", b"\n")

def digest(path: pathlib.Path) -> str:
    return hashlib.sha256(canonical_bytes(path)).hexdigest()[:16]

def build() -> dict[str, str]:
    out: dict[str, str] = {}
    for root in ROOTS:
        base = pathlib.Path(root)
        if not base.exists():
            continue
        for p in sorted(base.rglob("*")):
            if p.is_file():
                out[p.as_posix()] = digest(p)
    return out

def main() -> int:
    if "--write" in sys.argv:
        MANIFEST.write_text(json.dumps(build(), indent=2, sort_keys=True) + "\n")
        print(f"wrote {MANIFEST}")
        return 0
    current, recorded = build(), json.loads(MANIFEST.read_text())
    added = sorted(set(current) - set(recorded))
    removed = sorted(set(recorded) - set(current))
    changed = sorted(k for k in set(current) & set(recorded) if current[k] != recorded[k])
    for label, keys in (("ADDED", added), ("REMOVED", removed), ("CHANGED", changed)):
        for k in keys:
            print(f"{label}\t{k}")
    return 3 if (removed or changed) else 0

if __name__ == "__main__":
    raise SystemExit(main())

Exit 3 is deliberate: review required, not broken. CI turns 3 into a failure unless the PR carries the fixture-change-approved label. A plain 1 would get conflated with “script crashed” in the runner UI, and reviewers stop trusting it.

Step 2 — Log which fixture the loader actually read

A manifest alone cannot see shadowing or runtime generation. Record every resolution.

# tests/conftest.py
import json, os, pathlib
from tests.support import loader  # your repo's fixture loader

LEDGER = pathlib.Path(os.environ.get("FIXTURE_LEDGER", ".artifacts/fixture_ledger.jsonl"))
_resolve = loader.resolve   # returns a repo-relative posix path; adapt if yours returns absolute
_load = loader.load_fixture

def _recording_load(name, *args, **kwargs):
    LEDGER.parent.mkdir(parents=True, exist_ok=True)
    record = {
        "name": name,
        "path": _resolve(name),
        "test": os.environ.get("PYTEST_CURRENT_TEST", ""),
    }
    with LEDGER.open("a") as fh:
        fh.write(json.dumps(record) + "\n")
    return _load(name, *args, **kwargs)

loader.load_fixture = _recording_load

def pytest_sessionfinish(session, exitstatus):
    if not LEDGER.exists():
        return
    manifest = json.loads(pathlib.Path("tests/fixture_manifest.json").read_text())
    unhashed, by_name = set(), {}
    for line in LEDGER.read_text().splitlines():
        rec = json.loads(line)
        if rec["path"] not in manifest:
            unhashed.add(rec["path"])
        by_name.setdefault(rec["name"], set()).add(rec["path"])
    shadowed = {n: p for n, p in by_name.items() if len(p) > 1}
    if unhashed:
        print("UNHASHED FIXTURE LOADS:", sorted(unhashed))
    for name, paths in shadowed.items():
        print(f"SHADOWED FIXTURE {name}: {sorted(paths)}")
    if unhashed or shadowed:
        session.exitstatus = 1

Two failure classes fall out immediately. An unhashed load means the patch created data outside the declared roots. A shadowed name means one logical fixture has two physical sources — and which one wins depends on search order, not on review.

Step 3 — Bind a property check to each fixture

A hash tells you that data moved. It cannot tell you whether the move was legitimate. That needs an executable invariant.

# tests/properties/test_orders_fixture_contract.py
import json, pathlib

FIXTURE = pathlib.Path("tests/fixtures/orders.json")

def test_orders_fixture_shape():
    rows = json.loads(FIXTURE.read_text())
    assert rows, "fixture must not be empty"
    assert all({"id", "status", "total_cents"} <= set(r) for r in rows)
    assert any(r["status"] == "refunded" for r in rows), "refund path loses coverage"
    assert all(isinstance(r["total_cents"], int) for r in rows)

The gate now requires companion evidence. If tests/fixtures/** changes, tests/properties/** must change in the same commit — and the property suite must fail against the pre-patch fixture. Revert the fixture, keep the property: if it still passes, the property is decoration.

# tools/fixture_companion_check.sh
set -eu
changed=$(git diff --name-only "origin/main...HEAD")
if echo "$changed" | grep -q '^tests/fixtures/'; then
  if ! echo "$changed" | grep -q '^tests/properties/'; then
    echo "fixture changed with no property change"
    exit 1
  fi
fi

Decision table

Step 4 — Calibrate before you enforce

An uncalibrated gate becomes a rubber stamp within two weeks. Seed bugs into source, let a patch generator produce candidates, and record two numbers: how many fixture-touching patches the gate blocks, and how many legitimate fixture regenerations it wrongly blocks.

That calibration loop is where I used MonkeyCode's free model access to generate candidate patches against the seeded bugs, and its free server option to soak the property suite long enough to see whether the loader ledger recorded anything unexpected. Both availability claims are operator-supplied; I am not claiming model names, quotas, hardware, or permanence, and you should check current terms before depending on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the loop small: twenty seeded bugs and twenty candidate patches is enough to see whether the companion rule fires at all. If it never fires, your property files are probably too coarse to distinguish one fixture from another.

Limitations, and who should skip this

• Golden-file repos. Intentional regeneration is normal; the companion rule floods reviewers. Scope it to inputs, not outputs.
• Binary fixtures. Canonicalization is trivial but diffs are opaque. You get provenance without explanation — better than nothing, weaker than you want.
• Large fixture trees. Hashing is linear in bytes. On a multi-gigabyte tree, cache the manifest by path and mtime instead of rehashing in every job.
• Loader indirection. The ledger only sees loads that go through the wrapped function. Direct open() calls need an audit hook or a lint rule; do not pretend the ledger is complete.
• Short-lived prototype repos. The manifest churns more than the code. Add it when fixtures outnumber tests, not before.

The honest framing: this gate does not make agent patches safe. It makes one specific lie — the data moved, so the test passed — expensive to tell. That is a narrow win, and it is the one I could actually verify.

If you want to run the calibration loop on your own repo, the scripts above are the whole artifact; swap ROOTS and loader.resolve first, then measure the two block rates before you enable the CI check.