空白日誌如何通過發佈閘門：一次可重現失效模式的復盤

Postmortem: An Empty Log File Passed the Release Gate

The gate passed because the log file was missing. Nothing failed loudly. The pipeline read absence as success, promoted the artifact, and moved on.

This postmortem covers one failure class: fail-open verification on ephemeral compute. It applies to any pipeline that mixes short-lived workers with release decisions.

Framing note: the timeline below reconstructs a reproducible failure class, not a report from a named production system. The harness is included so you can run it yourself.

Impact

• A truncated generation run was promoted as complete.
• The release record claimed verification that never ran.
• Detection came later, through an unrelated diff review.

The damage was not a bad build. The damage was a false record. Later readers trusted that record.

Timeline

• T+0 — The release gate is triggered on a free, ephemeral worker.
• T+1s — The producer starts streaming JSON lines to run.log.
• T+4s — The worker is recycled mid-stream. No done event is written.
• T+6s — The verifier opens run.log, finds no FAIL string, returns true.
• T+7s — The artifact is promoted and stamped verified.
• T+2d — A reviewer notices the run had three of five expected steps.

Two details made this silent. The producer never flushed a completion marker. The verifier treated an empty read as an empty failure set.

The minimal reproduction

Three files reproduce it. The producer emits JSON lines; the gate decides pass or fail.

# producer.py — run on a short-lived worker
import json, sys, time

def emit(obj):
    print(json.dumps(obj), flush=True)

emit({"event": "start", "run": "R-1"})
for i in range(5):
    emit({"event": "step", "i": i, "run": "R-1"})
    time.sleep(0.2)
emit({"event": "done", "run": "R-1", "steps": 5})

Kill the process after two seconds. The file exists, is valid partially, and contains no done event.

The broken gate is four lines long:

# gate_bad.py — fail-open
from pathlib import Path

def gate(log_path: str) -> bool:
    text = Path(log_path).read_text() if Path(log_path).exists() else ""
    return "FAIL" not in text

This returns True for a missing file, an empty file, and a truncated file. Only one of those is a real pass.

Contributing factors

1. Fail-open defaults. A missing input was mapped to success instead of an error.
2. Shared failure domain. The producer and the verifier ran on the same recycled instance.
3. Unbound evidence. The log had no run id, no digest, and no verifier version.
4. Exit-code trust. The pipeline used the worker exit code, which a killed worker may not set cleanly.
5. No completion marker. Streaming output had no terminal event to require.
6. Free-tier recycling. Ephemeral instances get preempted. That is normal, not a bug.

Factor six is the one people blame first. It is also the least interesting. Preemption is a known property of short-lived compute.

Why the free tier widened the blast radius

Free model access and a free server option change the economics of CI, not the semantics of proof. Cheap generation invites more runs, more parallel branches, and more partial output.

MonkeyCode is one project that advertises free model access and a free server option alongside its open-source tooling; an advertised free allowance of 10M tokens was listed at the time of writing, and such figures change, so check the project's current terms yourself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The engineering point is narrower than the product. Free compute is fine for producing candidate artifacts. It is a poor place to certify them, for two reasons:

• The verifier inherits preemption, so it can die before it writes a verdict.
• The verifier shares a fate with the producer, so one outage corrupts both sides.

That is a placement problem, not a pricing problem.

The durable fix: content-addressed receipts

The gate must fail closed and bind to a receipt. A receipt is a small record, written by the verifier, that names what was checked.

# gate_fixed.py — fail-closed with a receipt
import hashlib, json
from pathlib import Path

REQUIRED = {"start", "done"}

def sha256(p: Path) -> str:
    return hashlib.sha256(p.read_bytes()).hexdigest()

def verify(log_path: Path, run_id: str) -> tuple[bool, str]:
    if not log_path.exists():
        return False, "receipt-missing"
    raw = log_path.read_bytes()
    events = []
    for line in raw.decode("utf-8", "replace").splitlines():
        if not line.strip():
            continue
        try:
            events.append(json.loads(line))
        except json.JSONDecodeError:
            return False, "receipt-truncated"
    if not REQUIRED <= {e.get("event") for e in events}:
        return False, "receipt-incomplete"
    if any(e.get("run") not in (run_id, None) for e in events):
        return False, "receipt-run-mismatch"
    return True, f"ok:{sha256(log_path)[:12]}"

The verifier then writes one record per decision:

{
  "run_id": "R-1",
  "artifact_sha256": "9f2c...",
  "log_sha256": "41ab...",
  "verifier": "gate_fixed.py@1.3.0",
  "verdict": "pass",
  "checked_at": "2026-10-15T00:00:00Z"
}

Three properties matter. The verdict is a file, not a log line. The verdict names the artifact digest. The verdict records which verifier version produced it.

Test plan

Run these five cases against any gate before trusting it.

1. Delete the log file. The gate must return receipt-missing.
2. Truncate the last line mid-JSON. The gate must return receipt-truncated.
3. Remove the done event. The gate must return receipt-incomplete.
4. Replay another run's log. The gate must return receipt-run-mismatch.
5. Run the happy path twice. The digests must match exactly.

Case four is the one most gates fail. Case one is the one that caused this incident.

Placement decision table

Limitations

The receipt pattern does not prove correctness. It proves that a specific verifier ran on a specific byte sequence. If the verifier is wrong, the receipt certifies the wrong thing.

It also adds a write step. On a preemptible worker, that write can fail. Move the receipt write to a durable runner, or accept a failed run and retry.

Who should not use this approach

• Teams whose evidence must stay in a specific region, with a contract behind it.
• Pipelines where a job must run longer than the free instance lives.
• Anyone who cannot afford a durable step to hold the verdict file.
• Teams that will not run the five cases above. A fail-open gate with a receipt stamp is worse than none.

What changed

One rule ended the failure class: no verdict, no promotion. The gate now returns an error code, not a boolean default. The producer and verifier run in different failure domains.

If you run generators on free compute, the receipt schema is the part worth arguing about. Send yours if it differs.