Agent 補丁新增時鐘等環境輸入，作者提出逐測試帳本追查流程

Most "flaky" agent patches are not flaky tests. They are tests that quietly started reading the clock, the RNG, the network, or an environment variable the base branch never touched — and then the rerun button erased the evidence. The fix is not more reruns. It is a per-test diff of ambient inputs between base and head, taken before merge, plus a quarantine that is keyed to the channel instead of the test so it cleans itself up.

Below is the workflow I use: a pytest plugin that records ambient channels, a gate that diffs two ledgers, a decision table for the verdict, and a null control that tells you when the ledger itself is lying.

1. Why reruns delete the diagnosis

A retry loop answers one question — "did it pass eventually?" — and destroys the more useful one: "what did this test read that it did not read before?" Once a test is retried, the failing input is gone, and the patch that introduced the input is now indistinguishable from a genuinely flaky neighbor.

The asymmetry matters. Reverting a patch is cheap. Debugging a test that fails 1 in 40 runs on shared CI, three weeks later, is not. So the ledger should be cheap and mandatory, and reruns should be the exception you have to justify.

2. Step 1 — Record ambient channels per test

Ambient input is anything a test reads that the patch can change without changing the test file: time.time(), random.random(), socket.socket.connect, os.getenv. Wrap the entry points, record a channel name per test, and write one JSON line per test call.

# conftest.py — reference implementation (Python 3.8+, pytest 7+)
import json, os, pathlib, random, socket, time
import pytest
from pytest import StashKey

LEDGER = pathlib.Path(os.environ.get("AMBIENT_LEDGER", ".ambient/ledger.jsonl"))
CHANNELS = StashKey()

@pytest.fixture(autouse=True)
def ambient(request, monkeypatch):
    seen = set()
    def mark(channel):
        seen.add(channel)

    t_time, t_time_ns = time.time, time.time_ns
    monkeypatch.setattr(time, "time", lambda: (mark("clock"), t_time())[1])
    monkeypatch.setattr(time, "time_ns", lambda: (mark("clock"), t_time_ns())[1])

    r_random = random.random
    monkeypatch.setattr(random, "random", lambda: (mark("rng"), r_random())[1])

    e_getenv = os.getenv
    monkeypatch.setattr(
        os, "getenv", lambda key, default=None: (mark(f"env:{key}"), e_getenv(key, default))[1]
    )

    c_connect = socket.socket.connect
    def connect(self, address):
        mark("net")
        return c_connect(self, address)
    monkeypatch.setattr(socket.socket, "connect", connect)

    yield
    request.node.stash[CHANNELS] = seen

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if report.when != "call":
        return
    LEDGER.parent.mkdir(parents=True, exist_ok=True)
    with LEDGER.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps({
            "test": item.nodeid,
            "outcome": report.outcome,
            "channels": sorted(item.stash.get(CHANNELS, set())),
        }) + "\n")

The output is append-only and boring, which is the point:

{"test": "tests/test_scheduler.py::test_retry_backoff", "outcome": "passed", "channels": ["clock", "env:RETRY_JITTER"]}

3. Step 2 — Diff base against head on the same machine

The comparison is only valid when the interpreter, pytest version, installed plugins, and collection order are identical. Any drift in those shows up in the diff as phantom channel changes, so pin the run:

# base tree, then head tree, same box, same command
AMBIENT_LEDGER=.ambient/base.jsonl pytest -p no:randomly -q
AMBIENT_LEDGER=.ambient/head.jsonl pytest -p no:randomly -q
AMBIENT_LEDGER=.ambient/head2.jsonl pytest -p no:randomly -q   # null control

That last line is the part people skip. Run head twice and diff head against head2. If the same test reports different channels on two identical runs, the ledger is unstable for that test — usually a thread, a forked subprocess, or a plugin that monkeypatches after your fixture — and no verdict from the base-vs-head diff is trustworthy until that noise is gone.

This is also where the run target stops being a detail. Because base and head must share a box, I keep one clean machine dedicated to ledger runs rather than reusing whatever laptop has the branch checked out. On the patch-generation side I use MonkeyCode's free model access to produce candidate patches, and I have been using its free server option as that dedicated run target, which keeps the two ledgers on identical ground. Those two availability claims are operator-supplied (free model access, free server option); I am not asserting specs, quotas, or hardware beyond that. The gate below is model-agnostic — it does not care which tool wrote the patch.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Another note: datetime.datetime.now() reads the C-level clock and will not be caught by patching time.time. For code that goes through datetime, add a freezegun or libfaketime layer, or accept that the ledger has a blind spot there — and say so in the PR.

4. Step 3 — Gate the diff with a decision table

# ambient_gate.py — reference; outputs one verdict row per changed test
import collections, json, pathlib, sys

ALLOWED_ENV = {"CI", "PYTHONHASHSEED"}   # keys whose presence is not a finding

def load(path):
    out = collections.defaultdict(lambda: {"channels": set(), "runs": 0, "fails": 0})
    for line in pathlib.Path(path).read_text(encoding="utf-8").splitlines():
        row = json.loads(line)
        slot = out[row["test"]]
        slot["channels"] |= set(row["channels"])
        slot["runs"] += 1
        slot["fails"] += row["outcome"] == "failed"
    return out

def verdict(test, added, slot):
    flaky = slot["fails"] > 0
    for channel in added:
        if channel == "net" and "/integration/" not in test:
            return "block", f"unit test opened a socket: {channel}"
        if channel.startswith("env:") and channel[4:] not in ALLOWED_ENV:
            return "needs-freeze", f"undeclared env read: {channel}"
        if channel in {"clock", "rng"} and flaky:
            return "needs-freeze", f"{channel} added and test failed in head"
        if channel in {"clock", "rng"}:
            return "probe", f"{channel} added, no failure yet"
    return "allow", "channel removed or benign"

base, head = load(sys.argv[1]), load(sys.argv[2])
bad = 0
for test in sorted(set(base) & set(head)):
    added = head[test]["channels"] - base[test]["channels"]
    if not added:
        continue
    code, why = verdict(test, added, head[test])
    bad += code == "block"
    print(f"{code:12} {test}  {why}")
sys.exit(1 if bad else 0)

The table it encodes:

5. Step 4 — Key the quarantine to the channel, not the test

A quarantine that names a test tends to outlive its cause, because nobody remembers why it exists. Key it to (test, channel) instead, and let the ledger retire it automatically: when the head ledger no longer reports that channel for that test, the freeze is stale and the gate fails until someone deletes it.

[
  {
    "test": "tests/test_scheduler.py::test_retry_backoff",
    "channel": "env:RETRY_JITTER",
    "reason": "head starts reading RETRY_JITTER; test asserts a fixed backoff",
    "owner": "platform",
    "issue": "#4821",
    "expires": "2026-11-03"
  }
]

Three rules make this survivable. First, a frozen test must not be retried (--reruns 0 for that node) — retries are exactly what hides the signal. Second, the freeze must carry an owner and an issue, or it is a comment with extra steps. Third, expiry is checked against the ledger rather than a calendar alone: after expires, the gate reports stale if the channel is gone, and overdue if it is not.

6. Step 5 — Probe determinism only where the ledger points

A probe verdict means the patch added a nondeterministic read but no failure was observed. Do not run the whole suite 50 times to be safe. Run that node, with shuffled order and varied seeds, and keep it to a small budget:

pytest tests/test_scheduler.py::test_retry_backoff -q --count=20 -p randomly

If it fails at least once, promote it to needs-freeze. If it survives the probe, allow the merge and leave a note in the PR naming the new channel — future readers will want to know the test now depends on wall-clock time.

7. Limitations and who should not use this

• Blind spots. The plugin only sees reads that go through patched entry points. C-extension clocks, datetime.now(), subprocesses, and most thread pools escape it.
• Noise from libraries. Patching os.getenv marks internal library reads too; on large dependency trees the allowlist becomes the real work.
• Infra flakiness is out of scope. If your failures come from shared databases, port collisions, or a saturated runner, the ledger will show no channel change and the gate will stay silent. That is correct behavior, not a bug.
• Cost. One JSON line per test call is cheap; on a 20k-test suite, batch the writes or sample.
• Skip it if your suite is mostly end-to-end against real services — ambient inputs are the design there, and blocking on them would block everything.
• Skip it if you cannot pin a stable run target. An unstable machine turns the ledger into a random number generator with extra output.

8. What to do next

Pick ten tests that have been rerun in the last month, add the fixture, and diff base against head twice. If the two head runs disagree, fix that before reading any verdict — the null control is the cheapest bug you will find all quarter.

If you want to try the split I described — free model access for candidate patches and the free server option as the ledger run target — start with the null control on a throwaway branch, not on your main suite. The gate only pays for itself once the ledger is stable enough to be believed.