Agent 補丁測試如何以確定性 manifest 管理非確定性輸入

A green run you cannot replay is a timestamp, not a signal. Agent-written patches hit this constantly: the new test reads the wall clock, opens a socket to a local stub, or depends on the interpreter's per-process hash seed. It passes once. On the next run — different second, different port, different PYTHONHASHSEED — it fails, and the failure gets filed as flakiness when it is really an undeclared input.

The common response is to quarantine the test. That treats the symptom, and quarantines have a way of becoming permanent. The response I use instead is to make the nondeterminism surface explicit before the patch is judged: every entropy source a test can touch must be named in a manifest, and a guard fails the test the moment it touches one that isn't.

Worth separating two ideas that get merged in review. A flake budget tells you how much variance you are willing to tolerate. A manifest tells you where the variance comes from in the first place. Budgets are policy; manifests are evidence. This article builds the manifest and the enforcement around it.

Three classes of nondeterminism, one file

The manifest is small on purpose. If it grows past a screen, you are documenting application structure, not test inputs.

Step 1 — Write the manifest before the test

Place nondeterminism.toml next to pyproject.toml. It is reviewed like code and owned by a human, not generated and merged unread.

# nondeterminism.toml
[clock]
frozen_at = "2026-01-01T00:00:00Z"
# tests may call these, but only through the frozen shim
allowed = ["time.time", "time.monotonic"]

[entropy]
seed = 20261016

[io]
allow_network = false
allow_subprocess = false
declared_reads = ["tests/fixtures/orders.jsonl"]

declared_reads uses repo-relative paths. A test that reads anything else gets flagged. That single rule catches most "works on my machine" fixture leakage, because agent patches routinely invent a fixture file locally and never commit it.

Step 2 — Arm the audit hook only during test bodies

Python 3.8+ exposes sys.addaudithook, which fires on events like open, socket.connect, and subprocess.Popen. Two constraints shape the design: an audit hook cannot be removed once installed, and pytest itself opens plenty of files during collection. So the hook is installed once and gated by a flag that is only true while a test body runs.

# tests/conftest.py
from __future__ import annotations

import json
import sys
import time
from pathlib import Path

import pytest

try:
    import tomllib  # Python 3.11+
except ModuleNotFoundError:  # pragma: no cover
    import tomli as tomllib

ROOT = Path(__file__).resolve().parents[1]
MANIFEST_PATH = ROOT / "nondeterminism.toml"
MANIFEST = tomllib.loads(MANIFEST_PATH.read_text())

DECLARED_READS = {
    str((ROOT / p).resolve()) for p in MANIFEST.get("io", {}).get("declared_reads", [])
}
FROZEN_EPOCH = 1767225600.0  # 2026-01-01T00:00:00Z

_state = {"armed": False}
_violations: list[tuple[str, str]] = []


def _audit(event: str, args: tuple) -> None:
    if not _state["armed"]:
        return
    if event == "socket.connect":
        _violations.append(("network", repr(args[1])))
    elif event == "subprocess.Popen":
        _violations.append(("subprocess", repr(args[0])))
    elif event == "open":
        target = args[0]
        if isinstance(target, int):  # already-open file descriptor
            return
        resolved = str(Path(str(target)).resolve())
        if resolved not in DECLARED_READS:
            _violations.append(("open", resolved))


sys.addaudithook(_audit)


@pytest.fixture(autouse=True)
def determinism_guard(monkeypatch):
    _violations.clear()
    monkeypatch.setattr(time, "time", lambda: FROZEN_EPOCH)
    monkeypatch.setattr(time, "monotonic", lambda: FROZEN_EPOCH)
    import random

    random.seed(MANIFEST["entropy"]["seed"])
    _state["armed"] = True
    try:
        yield
    finally:
        _state["armed"] = False
    if _violations:
        raise AssertionError("undeclared nondeterminism: " + json.dumps(_violations))

Run it plainly; no plugin ordering games are needed:

python -m pytest -q tests/

The code above is a reference implementation, not something I executed against your repository. Treat it as a starting point and adapt the declared-reads set.

Step 3 — Freeze the clock, and be honest about the gaps

The monkeypatch lines freeze time.time and time.monotonic. Two caveats matter more than the patch itself.

First, datetime.datetime.now() does not call Python-level time.time(). In CPython it goes through clock_gettime in C, so patching the time module leaves it untouched. If your patch uses datetime.now() for TTL logic, either inject a clock callable or use a dedicated freezing library. A guard that silently misses the most common clock source is worse than no guard, because it produces confidence.

Second, freezing monotonic to a constant can hang libraries that compute timeouts as deadline - monotonic(). If your dependency tree does that, make monotonic advance by a fixed step per call instead of returning a constant, and document the step in the manifest.

Step 4 — One declared seed per property check

Property checks are the cheapest way to make an agent patch admit what it assumes — but only if their sampling is reproducible. Derive the per-test seed from the manifest seed plus the node id, so a single manifest value still reproduces the whole suite, and two property checks do not share the same sample stream.

# tests/test_ledger_properties.py
import random
import zlib

import pytest

from app.ledger import apply_batch, signed_sum


@pytest.fixture
def rng(request):
    base = 20261016
    return random.Random(base ^ zlib.crc32(request.node.nodeid.encode()))


def test_signed_sum_is_conserved(rng):
    for _ in range(200):
        batch = [rng.randint(-10_000, 10_000) for _ in range(rng.randint(1, 40))]
        expected = sum(batch)
        assert signed_sum(apply_batch(batch)) == expected

The property is the assertion; the seed is bookkeeping. When this fails, the seed and the node id in the failure output are the entire reproduction recipe.

Step 5 — Emit a digest, but keep it out of the gate

A digest binds the manifest and the collected test set to a specific green run. It proves which inputs produced the pass. It does not prove the pass means anything.

# tools/determinism_digest.py
import hashlib
import pathlib
import subprocess
import sys

manifest = pathlib.Path("nondeterminism.toml").read_bytes()
collected = subprocess.run(
    [sys.executable, "-m", "pytest", "--collect-only", "-q"],
    capture_output=True,
    text=True,
    check=True,
).stdout.splitlines()

h = hashlib.sha256(manifest)
for line in sorted(line for line in collected if "::" in line):
    h.update(line.encode())
print(h.hexdigest()[:16])

Store the value next to the test report. Do not make it a merge condition — digests change whenever you add a test, and a gate that fires on every legitimate addition gets disabled within a week.

Step 6 — What this actually catches

Three failures I have seen this guard surface on agent patches, in roughly this order of frequency:

1. An undeclared fixture read. The test opens tests/fixtures/tmp_orders.json that was never committed. Locally it passes forever; on CI it fails, and the failure looks like a path bug.
2. A real socket. A stub was supposed to be injected, but the patch left a requests.get on a branch that only the new test exercises. allow_network = false turns a silent external dependency into a test failure with an address attached.
3. Hash-order dependence. The test iterates a set and asserts on the first element. It passes on a machine where PYTHONHASHSEED happens to be stable and fails when the harness sets it explicitly.

Note that case 3 is not fixed by freezing anything. The right repair is to sort the collection before asserting — the guard's job is only to make the defect arrive consistently.

When to require a manifest: decision table

Where free model access and a free server fit

Writing the first manifest draft by reading a diff is tedious and low-judgment, which is exactly the shape of work worth delegating. I paste the diff into a model, ask it to list every clock read, entropy source, and outbound call it can see, and paste the result into nondeterminism.toml. Then a human confirms each line — the model reliably catches time.time() and requests, and reliably misses entropy inside C extensions, dynamic imports, and anything reached through a plugin registry.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In that loop, MonkeyCode's free model access covers the drafting step at no cost, and the free server option is enough to run this harness because the workload is a Python test suite, not a model. Both are operator-supplied availability claims — check the current terms before you build a process around them, and keep the manifest file in your repo so switching endpoints is a one-line change.

Running the harness on the same host as the drafting is fine for iteration. It is not evidence: a determinism guard cannot validate itself on a single machine, because a machine with stable PYTHONHASHSEED and a warm fixture cache will hide exactly the class of defect you are hunting.

Limitations and who should skip this

• C extensions and native code bypass audit hooks. If your patch exercises compiled parsers, database drivers, or ML runtimes, the hook sees the call boundary and not the internal entropy. You need instrumented builds or an environment-level sandbox instead.
• Per-test overhead is nonzero. The hook allocates a resolved path per open and the audit callback runs in-process on every matching event. Measure it on your suite before enabling repo-wide; on I/O-heavy suites the cost is visible.
• Frozen monotonic can deadlock timeouts. Retry loops and connection pools that compute deadlines from monotonic will not make progress. Use a stepping clock or exempt those tests explicitly.
• This is not a correctness gate. A manifest says nothing about whether the assertions are meaningful. A patch with perfectly declared entropy and a tautological assertion still passes — that is a separate check, and it needs to run first.

Skip this approach if your tests are integration tests against real services by design, if you are on Python 3.7 or older (sys.addaudithook requires 3.8+), or if your team cannot commit to reviewing the manifest as code. A manifest that is auto-generated and merged unreviewed becomes a list of whatever the model noticed, which is precisely the false confidence the guard was supposed to remove.

Start with one module. Add the manifest, arm the guard, and run the suite twice with different PYTHONHASHSEED values. The second run is the one that tells you what the patch was actually assuming — and if you want to draft the first manifest without spending anything, the free model access is a reasonable place to begin.