48 小時實測筆記：故障測試可恢復批次作業，才放心使用免費伺服器

The first version of my batch pipeline ran end to end on the first attempt, and that is exactly why I distrusted it. A job that finishes once tells you nothing about what it does at item 37, when the process disappears and nobody is watching. So I spent 48 hours on the least glamorous work I know: build a resumable queue, then try hard to kill it.

These are field notes, in the format I keep coming back to because it forces me to write down the boring parts. Here is what I tried, what broke, and what I would repeat.

The setup, and the constraint that shaped it

The workload was small on purpose: a few dozen items, each one a model call plus a short transform, all written to disk as JSON. I ran it on the free server option and sent the calls through MonkeyCode's free model access, because the whole experiment had to be cheap enough to run badly five times.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The expensive part was never the model. It was my assumption that a long-running process on a machine I do not own will live long enough to finish. Ask yourself honestly: how many of your batch jobs would survive a restart at a random moment?

The resume contract: three states, one fenced write

I stopped thinking in terms of the job runs and started thinking in terms of a contract. Every item moves pending -> claimed -> done, and every transition is one transaction with a guard clause, so a stale worker cannot commit work it no longer owns.

The rules I settled on:

1. A worker may only claim items that are pending, or claimed with an expired lease.
2. A worker may only write done if it still holds the lease for that item, enforced inside the WHERE clause itself.
3. Side effects are keyed by item_id (out/{item_id}.json), so redoing work is wasteful but never duplicating.
4. Attempts are counted on claim, not on failure, because a killed process never gets to report a failure.

Rule 4 is the one I got wrong first, and it cost me most of an afternoon.

# ledger.py (stdlib only)
import os, sqlite3, time

DB = os.environ.get('LEDGER_DB', 'ledger.sqlite3')
LEASE = float(os.environ.get('LEASE_SECONDS', '90'))

def connect():
    con = sqlite3.connect(DB, isolation_level=None, timeout=30)
    con.execute('PRAGMA journal_mode=WAL')
    con.execute('''CREATE TABLE IF NOT EXISTS items (
        item_id TEXT PRIMARY KEY,
        state TEXT NOT NULL CHECK (state IN ('pending','claimed','done')),
        attempts INTEGER NOT NULL DEFAULT 0,
        lease_until REAL,
        worker TEXT,
        output TEXT)''')
    return con

def claim(con, worker, limit=1):
    con.execute('BEGIN IMMEDIATE')
    try:
        now = time.time()
        rows = con.execute('''SELECT item_id FROM items
            WHERE state = 'pending'
               OR (state = 'claimed' AND lease_until < ?)
            ORDER BY item_id LIMIT ?''', (now, limit)).fetchall()
        for (item_id,) in rows:
            con.execute('''UPDATE items
                SET state = 'claimed', attempts = attempts + 1,
                    lease_until = ?, worker = ?
                WHERE item_id = ?''', (now + LEASE, worker, item_id))
        con.execute('COMMIT')
        return [r[0] for r in rows]
    except Exception:
        con.execute('ROLLBACK')
        raise

def complete(con, item_id, worker, output_path):
    con.execute('BEGIN IMMEDIATE')
    try:
        cur = con.execute('''UPDATE items
            SET state = 'done', output = ?, lease_until = NULL
            WHERE item_id = ? AND state = 'claimed' AND worker = ?''',
            (output_path, item_id, worker))
        if cur.rowcount != 1:
            con.execute('ROLLBACK')
            raise RuntimeError('lease lost for ' + item_id)
        con.execute('COMMIT')
    except Exception:
        raise

That worker = ? inside the guard is the whole trick. Without it, a process that wakes up after its lease expired will cheerfully overwrite the result of the worker that took over. Have you ever written that UPDATE without the fence? I had, for years, and it never showed up until a machine restarted under me.

Crash injection: making failure reproducible

Real crashes are rare, which makes them a terrible test input. So I added a hook that kills the process at a chosen item, using os._exit so no finally block, no atexit handler, and no buffer flush gets a chance to run.

# worker.py (core loop)
import os, json, uuid
from ledger import connect, claim, complete

WORKER = uuid.uuid4().hex[:8]
CRASH_AT = int(os.environ.get('CRASH_AT', '0'))
con = connect()
done_here = 0

while True:
    ids = claim(con, WORKER, limit=1)
    if not ids:
        break
    item_id = ids[0]
    text = 'result for ' + item_id        # stands in for the model call
    with open(os.path.join('out', item_id + '.json'), 'w') as f:
        json.dump({'item_id': item_id, 'text': text}, f)
    complete(con, item_id, WORKER, 'out/' + item_id + '.json')
    done_here += 1
    if CRASH_AT and done_here == CRASH_AT:
        os._exit(9)                        # closest thing to kill -9

Then the test plan, which is the part I would keep forever:

python seed.py 50
for n in 0 1 7 19 33 49; do
  CRASH_AT=$n python worker.py
  python verify.py || echo "FAILED at CRASH_AT=$n"
done
python worker.py            # no hook: drain the queue
python verify.py

CRASH_AT=0 means no crash, so the last pass is a normal drain. If your ledger is correct, the queue finishes no matter where the earlier runs were killed, and the total attempts stay bounded.

verify.py checks three things, and each of them caught something real:

# verify.py
import os, sqlite3, sys
con = sqlite3.connect('ledger.sqlite3')
rows = con.execute('SELECT item_id, state, attempts FROM items').fetchall()
not_done = [r for r in rows if r[1] != 'done']
missing  = [r[0] for r in rows if not os.path.exists('out/' + r[0] + '.json')]
thrashing = [r for r in rows if r[2] > 3]
for label, bad in (('not done', not_done), ('missing output', missing), ('too many attempts', thrashing)):
    print(label + ':', bad)
sys.exit(1 if (not_done or missing or thrashing) else 0)

The too-many-attempts check is not about correctness; it is about whether my lease is shorter than my slowest item. If it fires, I tune the lease before I tune anything else.

What broke in the first 24 hours

The claim and the write were not in one transaction. My first version claimed in autocommit mode and wrote done in a second statement, so a kill between them left items claimed with no output. The fix was BEGIN IMMEDIATE around each transition, plus attempts counted on claim.

Expired claims were invisible until I looked for them. A killed worker leaves state='claimed' forever if nothing re-eligible it. My query said WHERE state = 'pending', so the queue looked empty while half the items sat in limbo. Adding lease expiry to the claim query made the queue self-heal.

Progress counted the wrong thing. I printed SELECT COUNT(*) FROM items and called it progress, which hit one hundred percent the moment I seeded the table. Counting state='done' is the only number that means anything to a human waiting on a run.

A file write outside the transaction. Writing out/x.json before the done update is safe only because the filename is keyed. The moment I appended to a shared results.jsonl instead, a restarted worker appended the same item twice. If your sink cannot be keyed, you need a dedupe table, not optimism.

When the free box is the right place for this

The pattern is simple: the free option is a good fit when losing the machine is cheap. It is a bad fit when losing the machine is the incident.

Limitations I will not pretend away

• SQLite is a single-writer store. Fine for one box, wrong for many workers over a network filesystem.
• Lease tuning stays guesswork until you measure. Too short and you double-process; too long and a crash costs you a full lease window of idle time.
• This gives you at-least-once with idempotent writes. It does not give you exactly-once, and nothing here is billing-grade accounting.
• If each item takes 200 milliseconds, none of this is worth the code. Restarting from zero is cheaper than the ledger.
• A resume contract protects you from wasted work, not from a loop that never terminates. That needs a separate spend or attempt cap.

And if your pipeline already runs inside a managed scheduler that retries and deduplicates for you, stop reading and go use it. Rebuilding this is only rational when you own neither the scheduler nor the hardware.

What I would repeat

1. Write the crash hook before the happy path. The happy path is the easy half and you already know it works.
2. Count attempts on claim, never on exception. Killed processes do not file bug reports.
3. Fence every terminal write with the worker identity, every single time, even when it feels redundant.
4. Make verify.py exit non-zero, so the loop above fails loudly instead of printing sad output into a scrollback nobody reads.
5. Run the worker under setsid so a dropped SSH session is not accidentally part of the test.

If you want to rehearse this on hardware you do not own, MonkeyCode's free server option and free model access are enough to run the exact test plan above. The test plan is the part that transfers, whichever machine you end up on.