48 小時批次處理實測：重試迴圈的代價高於模型本身

I had 120 short internal notes, one prompt template, and three variants I wanted to compare before Monday morning. Renting an always-on box for a weekend job felt silly, so I put the whole thing on a free server with free model access and walked away from it. What could possibly go wrong in 48 hours?

Turns out the model was never the problem. My retry loop was, and it took me two days to admit that.

Scope note before any numbers: every figure below is a parameter of my job — 120 items, 3 prompt variants, 4 attempts per item — not a benchmark of anyone's service. I am deliberately not publishing latency or throughput figures, because one weekend on one prompt mostly measures my prompt. Probe your own workload before you trust anything here.

What I was actually trying to do

• 120 short text notes, each needing a structured summary back as JSON.
• Three prompt variants, so I could diff the outputs before picking one.
• A hard constraint: no always-on VM for a weekend experiment, and no card on file for something I might abandon by Sunday.

My day-one plan was a for loop wrapping curl, which is the plan of someone who has never been rate limited. It has no checkpoint, no dedupe, and no idea how many requests it has already spent. Sixty lines later I had created all three problems at once.

What broke in the first 24 hours

1. My idempotency key had a timestamp in it

This is the line I am least proud of:

# Day one. Every retry is a brand-new job as far as the ledger is concerned.
key = f"{item['id']}-{int(time.time())}"

A key that changes on every attempt does not deduplicate anything — it just relabels duplicates. After the first crash-and-restart I had items with three result rows and no way to tell which one won. The fix is content addressing: hash the input and the parameters, never the clock.

2. A crash cost me the entire run, not one item

I was collecting results in a list and writing them out at the end of the loop. When the process died partway through, everything it had already paid for died with it. Append-only JSONL with a flush and an fsync per row is uglier to read and dramatically cheaper to lose.

3. I treated a 429 as "slow down a bit" instead of "read the header"

429 Too Many Requests may carry a Retry-After header, and when it does, that value is the answer. My loop ignored it and used its own constant delay, then hammered the endpoint at the same cadence as before. If the header is absent you fall back to exponential backoff with jitter — but jitter is not optional, or every worker in your fleet retries in lockstep.

4. I could not tell "slow" from "stuck"

Around hour 30 I had no idea whether the job was making progress or blocked on one item. That is not a model problem, it is a missing instrument. The thing I needed was not more logs, it was a small ledger: requests spent, items done, failures by class, attempts per success.

The runner I should have written on day one

The script below is the whole workflow. It is real and runnable; the one function you must replace is call_model, which is a stub so that nothing here pretends to know your client library.

#!/usr/bin/env python3
"""Resumable batch runner: content-addressed keys, append-only checkpoint, hard request budget."""
import argparse, hashlib, json, os, random, time

RETRYABLE = {"timeout", "connection", "rate_limited", "server_error"}
FATAL = {"auth", "bad_request", "content_filtered"}


def key_for(item_id, prompt_text, params):
    """Stable across retries and restarts. Changes only when the work changes."""
    payload = json.dumps({"id": item_id, "prompt": prompt_text, "params": params}, sort_keys=True)
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]


def load_done(path):
    done = {}
    if not os.path.exists(path):
        return done
    with open(path, "r", encoding="utf-8") as fh:
        for line in fh:
            line = line.strip()
            if not line:
                continue
            try:
                rec = json.loads(line)
            except json.JSONDecodeError:
                continue  # a killed process leaves at most one torn line
            done[rec["key"]] = rec
    return done


def append(path, rec):
    with open(path, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(rec, sort_keys=True) + "\n")
        fh.flush()
        os.fsync(fh.fileno())


def sleep_for(attempt, retry_after):
    if retry_after is not None:
        return min(float(retry_after), 60.0)
    base = min(30.0, 0.5 * (2 ** attempt))
    return base * (0.5 + random.random())  # jitter, always


def call_model(item, prompt_text, params):
    """Return (text, error_class, retry_after). error_class is None on success."""
    raise NotImplementedError("wire this to your model client")


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--items", required=True)
    ap.add_argument("--prompt", required=True)
    ap.add_argument("--out", default="results.jsonl")
    ap.add_argument("--max-requests", type=int, default=250)
    ap.add_argument("--max-attempts", type=int, default=4)
    ap.add_argument("--dry-run", action="store_true")
    args = ap.parse_args()

    prompt_text = open(args.prompt, encoding="utf-8").read()
    items = [json.loads(l) for l in open(args.items, encoding="utf-8") if l.strip()]
    params = {"variant": "v1"}
    done = load_done(args.out)
    todo = [it for it in items if key_for(it["id"], prompt_text, params) not in done]

    print(f"items={len(items)} done={len(done)} todo={len(todo)} budget={args.max_requests}")
    if args.dry_run:
        return

    spent = fatal = 0
    for item in todo:
        key = key_for(item["id"], prompt_text, params)
        for attempt in range(args.max_attempts):
            if spent >= args.max_requests:
                print("budget reached, stopping cleanly; rerun to resume")
                return
            spent += 1
            text, err, retry_after = call_model(item, prompt_text, params)
            if err is None:
                append(args.out, {"key": key, "id": item["id"], "attempts": attempt + 1, "text": text})
                break
            if err in FATAL:
                append(args.out, {"key": key, "id": item["id"], "error": err, "fatal": True})
                fatal += 1
                break
            if err in RETRYABLE and attempt + 1 < args.max_attempts:
                time.sleep(sleep_for(attempt, retry_after))
                continue
            break  # exhausted: no row is written, so the next run picks it up again
    print(f"requests={spent} fatal={fatal}")


if __name__ == "__main__":
    main()

The commands I now run in this order, every time:

# 1. Dry run first. It prints the work plan and spends nothing.
python3 batch_runner.py --items items.jsonl --prompt prompt.txt --dry-run

# 2. Real run, with a budget small enough to survive a surprise.
python3 batch_runner.py --items items.jsonl --prompt prompt.txt \
  --out results.jsonl --max-requests 250

# 3. The same command resumes. No flags change, no state to clean up.
python3 batch_runner.py --items items.jsonl --prompt prompt.txt \
  --out results.jsonl --max-requests 250

One consequence worth naming: because the prompt text is inside the key, editing the prompt invalidates every cached row. That is correct for correctness and expensive for iteration, so I keep the prompt in its own file and bump params["variant"] on purpose rather than on accident.

The ledger is the artifact that survives

The script is disposable. The file it appends to is the thing I read on Monday.

A five-line summary is enough to read it:

python3 - <<'PY'
import json, collections
rows = [json.loads(l) for l in open("results.jsonl") if l.strip()]
print(collections.Counter(r.get("error", "ok") for r in rows))
print("avg attempts:", sum(r.get("attempts", 0) for r in rows) / max(len(rows), 1))
PY

Where the free model and free server actually fit

This is the part where the weekend budget mattered. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I ran the batch against MonkeyCode's free model access and left the runner on its free server option, which meant the experiment cost me a weekend instead of a VM invoice plus per-token anxiety while I iterated on three prompt variants.

That is the honest extent of the claim. Free access is not a throughput guarantee, a permanence guarantee, or an SLA — treat any free tier as something that can be slower or busier than you hoped, and let the ledger tell you rather than your assumptions. And the usual rule applies regardless of provider: do not paste secrets or regulated data into an endpoint you have not reviewed for that purpose. Nothing about a free tier changes that.

When this approach is the right one

Who should not do this

• Anyone whose job must finish by a wall-clock deadline, because shared capacity makes a tail latency promise you cannot keep.
• Anyone processing data they cannot legally paste into a third-party endpoint, no matter how short the experiment.
• Anyone who will not read the ledger, because an unmeasured free tier is just an unbounded one.
• Teams that need an auditable retention story — a JSONL file on a temporary box is not that story.

What I would repeat, and what I would throw away

1. Repeat: the dry run. It costs nothing and it has caught a bad item file twice.
2. Repeat: content-addressed keys built from input plus parameters, never from time.
3. Repeat: a hard request budget, set deliberately low on the first pass.
4. Throw away: collecting results in memory until the end. That habit is only cheap when nothing ever fails.
5. Throw away: constant-delay retries. Header first, jittered backoff second, hope never.

If you take one thing from these 48 hours, take the ledger file rather than the runner — the runner gets rewritten, but the record of what your retries actually cost is the part that keeps answering questions. And if you do try the same batch on a free tier, run the --dry-run first; it is the cheapest 200 milliseconds in the whole workflow.