批次工作程序中斷，重跑又處理所有項目：48 小時實戰筆記

Two days ago I started a batch of prompt evaluations, closed the laptop lid, and assumed the free server would keep chewing through the queue while I slept. It did not. When I opened the terminal the next morning, the worker was gone and the only output was a JSONL file that stopped in the middle of a record. Why do we keep trusting long-running processes we never instrumented?

This is a field note about the 48 hours that followed: the naive loop I wrote, the four ways it lied to me, and the small ledger I now put in front of every batch that outlives a single shell session.

What I built first, and why it felt fine

The first version was about thirty lines long and it did exactly what the happy path needed. I pointed a client at MonkeyCode's free model access, ran the script on the free server option, and appended one JSON line per result. Nothing about that loop looked wrong at the time.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The caps and version strings in the code below are placeholders from my own config, not benchmark results or vendor limits; substitute your own numbers.

# naive_batch.py - the version that lost a night
import json, pathlib, urllib.request

items = [line.strip() for line in open('prompts.txt') if line.strip()]
out = pathlib.Path('results.jsonl')

for i, prompt in enumerate(items):
    body = json.dumps({'prompt': prompt}).encode()
    req = urllib.request.Request(ENDPOINT, data=body, headers={'Content-Type': 'application/json'})
    with urllib.request.urlopen(req, timeout=60) as resp:
        text = json.loads(resp.read())['text']
    with out.open('a') as f:            # append, no fsync, no key
        f.write(json.dumps({'i': i, 'prompt': prompt, 'text': text}) + '\n')

Read it again and count the assumptions. The index i is the only identity an item has, the open file handle is the only progress record, and the loop has no idea what it already finished. If the machine disappears, everything the process knew disappears with it.

What broke, in the order I noticed it

1. Progress lived in memory. The enumerate counter was the resume state, so a fresh process always started at zero and re-sent everything.
2. Retries had no identity. When one call timed out and I re-ran the loop, the same prompt produced a second row with no way to tell which row was the retry.
3. Writes were not atomic. A worker killed mid-write left a line that could not be parsed, and my summary script crashed on the last record instead of skipping it.
4. I could not separate queued from finished. A prompt that never started and a prompt that returned an empty answer looked identical in the file.

Which of those is a model problem? None of them. Every failure came from my bookkeeping, and that was the uncomfortable part of the 48 hours.

The fix: resume by content hash, not by hope

The new version gives each work item a key derived from the thing that actually determines the answer: the normalized prompt, the prompt version, and a model alias I control. Then it reads the ledger before it sends anything.

The ledger module

# ledger_batch.py - resume by content hash, not by hope
import hashlib, json, os, pathlib

LEDGER = pathlib.Path('ledger.jsonl')
PROMPT_VERSION = 'eval-v3'
MODEL_ALIAS = 'free-default'    # an alias you own, not a vendor model id
MAX_ITEMS_THIS_RUN = 40         # placeholder: pick from your own time budget
MAX_ATTEMPTS = 3

def normalize(prompt: str) -> str:
    return ' '.join(prompt.split()).strip().lower()

def key_for(prompt: str) -> str:
    raw = '\x1f'.join([PROMPT_VERSION, MODEL_ALIAS, normalize(prompt)])
    return hashlib.sha256(raw.encode('utf-8')).hexdigest()

def load_state() -> tuple[set[str], dict[str, int]]:
    ok, attempts = set(), {}
    if not LEDGER.exists():
        return ok, attempts
    for line in LEDGER.read_text(encoding='utf-8').splitlines():
        try:
            rec = json.loads(line)
            k = rec['key']
        except (json.JSONDecodeError, KeyError):
            continue          # a half-written tail line is expected after a hard kill
        if rec.get('status') == 'ok':
            ok.add(k)
        else:
            attempts[k] = attempts.get(k, 0) + 1
    return ok, attempts

def append(record: dict) -> None:
    line = json.dumps(record, sort_keys=True, ensure_ascii=False)
    with LEDGER.open('a', encoding='utf-8') as f:
        f.write(line + '\n')
        f.flush()
        os.fsync(f.fileno())   # the point is to survive the worker vanishing

def call_model(prompt: str) -> str:
    ...   # your client, your endpoint, your retry policy

The loop that uses it

def run(prompts: list[str]) -> None:
    ok, attempts = load_state()
    todo = [p for p in prompts if key_for(p) not in ok]
    todo = [p for p in todo if attempts.get(key_for(p), 0) < MAX_ATTEMPTS]
    print(f'{len(ok)} ok, {len(todo)} remaining, cap {MAX_ITEMS_THIS_RUN}')

    for prompt in todo[:MAX_ITEMS_THIS_RUN]:
        k = key_for(prompt)
        try:
            text = call_model(prompt)
        except Exception as exc:
            append({'key': k, 'status': 'error', 'error': type(exc).__name__})
            continue
        append({'key': k, 'status': 'ok', 'prompt': prompt, 'text': text,
                'prompt_version': PROMPT_VERSION, 'model_alias': MODEL_ALIAS})

Two properties matter more than the code style. First, re-running the script is cheap, because finished keys are filtered before any network call happens. Second, changing PROMPT_VERSION or MODEL_ALIAS makes every old key stale on purpose, so a tweaked prompt never inherits yesterday's answers.

Inspecting the ledger without a database

wc -l ledger.jsonl
jq -r 'select(.status=="ok") | .key' ledger.jsonl | sort | uniq -d    # should print nothing
jq -r 'select(.status=="error") | .error' ledger.jsonl | sort | uniq -c | sort -rn | head

The duplicate check is the one I run before I trust any summary. If uniq -d prints keys, some code path is appending twice, and that is a bug in the writer rather than a quirk of the model.

A decision table for when this is worth the ceremony

Does the table feel too cautious? Good. The ledger is cheap insurance for batches that span hours, and pure overhead for scripts that finish before your coffee does.

What I would repeat, and what I would change

• Repeat: the per-run cap. It turned an unbounded night into a predictable slice I could inspect in the morning.
• Repeat: prompt_version and model_alias inside the key, because they are the two edits that silently invalidate everything.
• Repeat: fsync on append. It is one line, and it changed a corrupted tail into a recoverable one.
• Change: move the ledger off the free server's disk. If the instance is replaced, a local ledger goes with it, so write to storage you control.
• Change: log the request start time, so I can distinguish a slow item from one that never started.
• Change: keep the prompt text in the ledger row. Reconstructing inputs from an index was a mistake I will not repeat.

Limits, and who should not do this

This pattern is not exactly-once delivery, and I want to be blunt about that. A crash between the model response and the fsync still costs one duplicate call on resume, and nothing here gives you ordering across multiple workers. The uniq -d check finds duplicates after the fact; it does not prevent them.

Free tiers are also not a contract. The free model access and free server option I used were fine for this batch, but availability, capacity, and terms can change, so treat both as replaceable and keep the ledger portable. Anyone reconciling money per request, running parallel consumers, or needing sub-second scheduling should use a real queue with leases and idempotency keys instead of a text file.

Who is this for, then? Developers running evaluation batches, backfills, or slow report jobs on infrastructure they do not fully control, and who would rather resume than restart. If that is you, run ten items first, watch the ledger grow, and only then let the batch run overnight.