建立 CLI 執行基準前，兩次執行結果必須一致

Freeze a repo's behavior only after two independent runs produce the same transcript. If two runs disagree, you have no baseline, only noise. This article builds a process-boundary harness that measures that, then guards the smallest safe change.

Most refactor baselines live at the function boundary. Someone calls build_report() and snapshots the returned string. That is convenient, and it hides the parts that actually break: exit codes, stderr, files written, and ordering across a whole run.

Why the Process Boundary Wins First

Functions change signatures freely. A CLI's contract changes slowly and visibly. Users script against exit codes and stdout, not against your internal call graph.

A transcript is four things: exit code, stdout, stderr, and the output tree. Digest the output files and you catch silent data corruption. Digest them with SHA-256 and you catch one flipped byte.

This is coarser than unit tests, and that is the point. You can record it before you understand the code. Understanding comes from reading the diff, not from reading the module.

Step 1: Pick One Seam and One Fixture

Choose the entry point your team actually runs. Often that is a bin/ script or a python -m package.cli invocation. Avoid library internals for the first recording.

Create one fixture directory per scenario. Copy in the minimum inputs: a small CSV, a config file, a SQLite file. The harness copies the fixture into a temp dir, so runs never share state.

Declare one output directory and require the command to write only there. Mixing outputs into the fixture root makes the digest useless. If the command writes everywhere, that itself is a finding worth recording.

Step 2: Build the Normalizer Registry

Some bytes are different on every run and carry no contract. Normalize them explicitly, with a named pattern per rule. Never normalize with a broad catch-all.

NORMALIZERS = [
    (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b"), "<uuid>"),
    (re.compile(r"\b\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?\b"), "<timestamp>"),
    (re.compile(r"(?:/tmp|/private/var)/[\w./-]+"), "<tmp>"),
    (re.compile(r"\b0x[0-9a-fA-F]{6,}\b"), "<addr>"),
    (re.compile(r"\b\d+(?:\.\d+)? ?(?:ms|s|KB|MB)\b"), "<duration>"),
]

Keep the registry short and reviewable. Every entry is a promise that this field is not part of the contract. Document the promise in a comment above the pattern.

Step 3: Make Determinism a Precondition

This is the step most harnesses skip. Run the command twice with different hash seeds. Compare the normalized transcripts.

env = dict(os.environ, PYTHONHASHSEED=seed)  # seed is "0" then "1"

Python randomizes string hashing per process by default; PYTHONHASHSEED is the documented override (docs). Two seeds expose dict-order and set-order leakage cheaply.

If the two runs disagree, do not record a golden file. Fix the order instead, which is usually a one-line change:

for row in sorted(rows, key=lambda r: r.customer_id):

Pinning the seed to 0 in production would hide the bug for real users. That trade is almost never worth it. Sort the data, then freeze.

Step 4: Record, Then Verify the Recording

python golden.py record --name invoice_cli --fixture fixtures/invoice -- ./bin/invoice --month 2026-09
python golden.py check  --name invoice_cli --fixture fixtures/invoice -- ./bin/invoice --month 2026-09

The harness below is the whole artifact, around sixty lines and no dependencies.

# golden.py
import argparse, hashlib, json, os, re, shutil, subprocess, tempfile
from pathlib import Path

GOLDEN = Path("golden")

def normalize(text):
    for pattern, repl in NORMALIZERS:
        text = pattern.sub(repl, text)
    return text

def digest_tree(root):
    rows = []
    for path in sorted(root.rglob("*")):
        if path.is_file():
            data = path.read_bytes()
            rows.append([str(path.relative_to(root)),
                         hashlib.sha256(data).hexdigest(), len(data)])
    return rows

def run_once(cmd, fixture, seed):
    work = Path(tempfile.mkdtemp(prefix="golden-"))
    shutil.copytree(fixture, work, dirs_exist_ok=True)
    env = dict(os.environ, PYTHONHASHSEED=seed)
    try:
        proc = subprocess.run(cmd, cwd=work, env=env, timeout=120,
                              capture_output=True, text=True)
        out = work / "out"
        return {
            "exit_code": proc.returncode,
            "stdout": normalize(proc.stdout),
            "stderr": normalize(proc.stderr),
            "files": digest_tree(out) if out.exists() else [],
        }
    finally:
        shutil.rmtree(work, ignore_errors=True)

def record(cmd, fixture, name):
    a, b = run_once(cmd, fixture, "0"), run_once(cmd, fixture, "1")
    if a != b:
        print("UNSTABLE: two runs disagree; do not freeze")
        return 1
    GOLDEN.mkdir(exist_ok=True)
    (GOLDEN / f"{name}.json").write_text(json.dumps(a, indent=2, sort_keys=True))
    return 0

def check(cmd, fixture, name):
    want = json.loads((GOLDEN / f"{name}.json").read_text())
    got = run_once(cmd, fixture, "0")
    if want == got:
        print("MATCH")
        return 0
    for key in ("exit_code", "stdout", "stderr", "files"):
        if want[key] != got[key]:
            print(f"DIFF {key}")
            print(f"  want {json.dumps(want[key])[:300]}")
            print(f"  got  {json.dumps(got[key])[:300]}")
    return 1

if __name__ == "__main__":
    p = argparse.ArgumentParser()
    p.add_argument("mode", choices=["record", "check"])
    p.add_argument("--name", required=True)
    p.add_argument("--fixture", required=True)
    p.add_argument("cmd", nargs=argparse.REMAINDER)
    a = p.parse_args()
    fn = record if a.mode == "record" else check
    raise SystemExit(fn(a.cmd, a.fixture, a.name))

Commit the golden JSON. Run check immediately after recording. A recorder that cannot reproduce its own snapshot is broken, and you want to learn that now.

Step 5: Make the Smallest Safe Change

Now touch one thing. Rename a variable, extract one function, move one import. Run check after each edit, not after five edits.

MATCH means the observable behavior survived. That is a narrow claim, and it is the claim you need before continuing. Repeat until the messy region is parsed into small, verified moves.

If you must change behavior deliberately, do it in a separate commit with its own golden update. Never mix a behavior change into a mechanical commit. Reviewers cannot separate the two afterward.

Step 6: Triage Every Diff With This Table

Two rules keep this honest. Never re-record a golden file to silence a diff you have not explained. And never widen a normalizer to cover a real behavior change.

A Concrete Failure I Hit

The first fixture I recorded printed a temp path in a warning line. The harness flagged UNSTABLE, not a diff, because the two seeds produced different paths. That was the harness working correctly.

I added the <tmp> rule, re-recorded, and got MATCH. The real bug appeared later, when the sorted output revealed duplicate rows my extraction step had introduced.

Where a Free Model and a Disposable Box Fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The normalizer registry is fiddly regex work. I paste a failing diff and ask a free model for candidate patterns, then test each candidate by hand. I keep only rules I can explain in one sentence, because an unexplained rule silently deletes evidence.

The second-seed run also needs a clean machine. A laptop with leftover files in the fixture path weakens the transcript. MonkeyCode's free server option is a reasonable place to run the gate, since a disposable box keeps local state out of the recording. I cannot promise specs, quotas, or permanence for that tier, so treat it as replaceable and pin your interpreter version in the golden file metadata.

If your repo already has a runnable entry point, record one scenario today. The free model and free server tiers are enough to try the loop before you commit tooling time.

Limitations and Who Should Skip This

This harness proves stability, not correctness. A wrong result that is reproducibly wrong will freeze happily. Pair it with assertions about values, not just about sameness.

Some noise resists normalization. Thread interleavings, wall-clock scheduling, and network responses will keep the UNSTABLE branch hot. Add a seeded or recorded-input mode to the code before you add more regexes.

Skip this approach if the command writes to a live database you cannot clone. Skip it if the output tree is large binary blobs, where hashing is slow and diffs are unreadable. Skip it if there is no entry point to run yet, because a harness cannot characterize code that never executes.

For everyone else, the sequence is fixed. Two runs must agree, then you freeze, then you make one small change and check again.