摘要批次工作在第 27 項崩潰後，如何避免重複呼叫

My digest worker died at 2:14 a.m. on item 27 of 40. The traceback was a timeout — unremarkable, the kind of thing a free endpoint does on a slow night. I restarted the script and watched it summarize items 1 through 26 all over again. Those summaries were already on disk, already correct, already paid for in tokens. I paid twice for the same twenty-six paragraphs because my script had no memory of ever having done them.

So here is the question I actually care about: can you make a batch worker that resumes instead of restarting, in a way that makes a flaky free tier boring? Yes — and the whole mechanism is one table with one column that decides whether a request is new or already answered. This is the case study of building it, including the version where my hash was wrong and I double-charged myself anyway.

What I wanted

Background first. I keep a reading list of paper abstracts (currently 40 entries, stored as JSON Lines), and I want one short summary per abstract so I can skim before a seminar. The summaries are cheap individually and annoying in aggregate, which is exactly the shape of a job you want to run overnight and never think about again.

Goal: run the worker, kill it at any moment, run it again, and pay for each unique abstract exactly once. Not "usually once." Exactly once, verifiable from a counter.

Prerequisites, exactly

Python 3.11.9 on Linux; sqlite3, hashlib, urllib, and http.server all from the standard library. No virtualenv needed, no third-party packages, no API key for the first half of this post. Check with python3 --version — anything from 3.10 up will do, but 3.11.9 is what I ran.

Step 0: keep the network out of the lesson

Testing idempotency against a real endpoint is a bad idea, because you cannot tell whether you doubled a call or lost one. So I wrote a short stub that counts requests and confesses the count in its reply.

# stub_endpoint.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

CALLS = {"n": 0}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        payload = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        CALLS["n"] += 1
        body = json.dumps({"summary": f"stub summary #{CALLS['n']}",
                           "chars": len(payload["text"])}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

When the output says summary #3, it means "this is the third request that ever arrived." That number is my ledger for the ledger.

Step 1: the worker

The worker below hashes each abstract, looks the hash up in SQLite, and only calls the endpoint on a miss. If an entry is missing because the process died, the next run fetches that entry and nothing else.

# digest.py
import hashlib, json, os, sqlite3, sys, urllib.request

DB, ENDPOINT = "digest.sqlite3", "http://127.0.0.1:8765/summarize"
CRASH_AT = int(os.environ.get("CRASH_AT", "-1"))

def normalize(text: str) -> str:
    return " ".join(text.split())

def key_for(text: str) -> str:
    return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()

def summarize(text: str) -> str:
    req = urllib.request.Request(
        ENDPOINT, data=json.dumps({"text": text}).encode("utf-8"),
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.loads(r.read())["summary"]

def main(path: str) -> None:
    con = sqlite3.connect(DB)
    con.execute("CREATE TABLE IF NOT EXISTS digests("
                "key TEXT PRIMARY KEY, source TEXT, summary TEXT)")
    rows = [json.loads(line) for line in open(path, encoding="utf-8")]
    for i, row in enumerate(rows, start=1):
        k = key_for(row["text"])
        hit = con.execute("SELECT summary FROM digests WHERE key=?", (k,)).fetchone()
        if hit:
            print(f"{row['id']} cached  {hit[0][:24]}")
            continue
        if i == CRASH_AT:
            print(f"{row['id']} CRASH  simulated")
            sys.exit(1)
        s = summarize(row["text"])
        con.execute("INSERT INTO digests(key, source, summary) VALUES (?,?,?)",
                    (k, row["id"], s))
        con.commit()
        print(f"{row['id']} fetched {s[:24]}")

main(sys.argv[1])

Run it, break it, run it again:

$ CRASH_AT=27 python3 digest.py readings.jsonl | tail -2
reading-026 fetched stub summary #26
reading-027 CRASH  simulated
$ python3 digest.py readings.jsonl | head -3
reading-001 cached  stub summary #1
reading-002 cached  stub summary #2
reading-003 cached  stub summary #3

On the second run the stub's counter stays at 26 until the loop reaches the missing entries. That is the property I wanted: a crash costs you the in-flight request, not the finished ones.

The input that broke my first version

My first hash was hashlib.sha256(row["text"].encode()).hexdigest() — the raw string, no normalization. Then I regenerated readings.jsonl from a PDF extractor, and one abstract came back with a trailing space and a soft-wrapped newline in the middle. Same paper. Different bytes. Different hash. The worker saw a stranger and paid for it again, and my "exactly once" claim quietly became "twice, if your source formatting ever moves."

The fix is normalize(), collapsing all whitespace before hashing, which is in the code above. It is not a general solution — it also treats two genuinely different strings that differ only in spacing as the same document. For abstracts that is fine. For a legal clause, or anything where whitespace carries meaning, hash a canonical ID instead and keep the text out of the key entirely.

To reproduce my bad input, duplicate one line and add " \n" to its text before running. With normalize() removed you will see a second fetched line for that paper. With it in place, cached.

Where the free tier fits

This is where the product lives, so let me be plain about it and about what I can and cannot check. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project, and its outreach material describes two things that matter to a student worker like mine: free access to models, on the order of a ten-million-token allowance, and a free server option. I cannot verify quotas or hardware from outside, and I have not audited the repository, so treat every one of those claims as something to test on your own account before you build a habit on top of it.

Two honest observations, though. First, free model access is what turns a 40-item digest into a Saturday afternoon project instead of a budget decision — but free access is also why the ledger matters more, not less. When requests feel free, it is tempting to skip idempotency and just re-run everything, and that is how a cheap habit becomes an expensive one the first time an allowance changes underneath you. Second, a free server option addresses the lid problem: my earlier version only ran while my laptop was open. A worker with a durable ledger that survives a closed lid is a different animal from a worker that survives only if you sit there and watch it.

If you want to try the loop end to end, keep the stub in the repo as a test fixture, point ENDPOINT at your own endpoint, and run a small batch first. That is the softest suggestion I can make, and it is genuinely the right order to do things in.

What this does not solve

The ledger is not a queue. It does not retry failures with backoff, it does not cap concurrency, and it will happily grind forever on a file that keeps growing. It also stores every summary uncompressed in SQLite — fine at 40 rows, less fine at 40,000.

Do not use this approach if you are sending material that should never reach a third-party endpoint at all, if you need a response-time guarantee, or if someone else depends on the job finishing tonight. A single-process script with a local database is a student tool, and it should stay one until the requirements say otherwise.

Predict, then test

Before you run anything, look at these two entries and tell me which one the un-normalized version double-charges:

{"id": "a", "text": "Attention is all you need."}
{"id": "b", "text": "Attention is all you need.\n"}

If you said "b," you are right for the boring reason, and now you know why normalize() sits in front of the hash rather than behind it. If you find a fixture where my normalization merges two documents that should stay apart — a code snippet where the indentation is the entire point, say — send me the minimal counterexample. That is the version of this post I would rather be reading.