LLM 呼叫成功率達 99%，100 次呼叫的任務仍有約 63% 失敗率

You write a script that summarizes a 100-page PDF, one LLM call per page. It dies on page 71 with a timeout. You run it again, and it starts from page 1. No single part is broken: the API had one bad second, which happens. But the rerun pays for pages 1 to 70 a second time, and its odds of dying are the same as the first run's.

One call is a coin toss, a job is a streak

Model APIs bill per token, a chunk of text about three-quarters of an English word. Input tokens are what you send, output tokens are what the model writes back. A call can fail with nobody at fault: a timeout, an overloaded server, an answer your code can't parse. Call the share of calls that succeed p. At p = 99%, a single call is safe enough.

A job finishes only if every call in it succeeds, so the odds multiply: a job of N calls finishes with probability p to the power N. Two calls at 99%: 0.99 × 0.99 ≈ 98%. A hundred calls: 0.99¹⁰⁰ = 36.6%, so the job fails in 63% of runs. At 95% per call, a hundred-call job finishes 0.6% of the time. Both rates are my assumptions, not measurements: 99% is a call that fails only on network trouble, 95% is closer to one whose answer must also parse as JSON. Short jobs barely notice: five calls at 99% still finish 95% of the time.

Starting over pays for finished pages again

What a failure costs depends on what your code does next. Start again from call 1, and every call before the failure is bought again, as many times as it takes. Retry only the call that failed, and nothing finished is paid for twice. Here are both, for jobs of 20 and 100 calls:

PRICE_IN, PRICE_OUT = 2.00, 12.00                   # GPT-5.6 list price, $ per 1M tokens
CALL = (1_000 * PRICE_IN + 250 * PRICE_OUT) / 1e6   # one page: $0.005

def job(n, p):
    finish = p ** n                                 # every call has to succeed
    start_over = (1 - finish) / ((1 - p) * finish)  # calls paid, restarting from call 1
    retry = n / p                                   # calls paid, retrying the failed call
    return finish, start_over, retry

for n, p in [(20, 0.99), (20, 0.95), (100, 0.99), (100, 0.95)]:
    finish, start_over, retry = job(n, p)
    print(f"{n:3} calls at {p:.0%}: finishes {finish:5.1%}, "
          f"start over ${start_over * CALL:5.2f}, retry ${retry * CALL:.2f}")

A page is 1,000 tokens in and 250 out. On GPT-5.6 at its list price, the one the vendor publishes, that's $2 per million input tokens and $12 per million output: $0.005 a page, $0.50 for the PDF. The prices come from the catalog of the LLM gateway I work on, read on 29 October 2026. The start_over line is the textbook formula for the expected wait until N successes in a row.

Each × compares with a run without errors: $0.10 for 20 pages, $0.50 for 100. Retrying costs the same small extra at any length, because you pay for N ÷ p calls. Starting over grows with the job. At 95%, the 100-page PDF takes 169 runs on average, and each failed run dies about 19 pages in: 3,358 paid calls, $16.79 for a $0.50 job. By hand you'd quit after a few runs, with nothing saved.

Retry the page, keep the pages

The fix has two parts. Retry the call that failed, a couple of times with a growing pause, instead of failing the whole job. And write each answer to disk as soon as you have it. Saved progress like this is a checkpoint: if the job dies anyway, the next run reads the file, skips finished pages and starts from the one that failed.

import json, pathlib, time

def run_job(pages, call_model, out=pathlib.Path("results.jsonl"), attempts=3):
    done = set()
    if out.exists():                      # an earlier run already paid for these
        done = {json.loads(line)["page"] for line in out.read_text().splitlines()}
    with out.open("a") as f:
        for i, page in enumerate(pages):
            if i in done:
                continue
            for attempt in range(attempts):
                try:
                    answer = call_model(page)
                    break
                except Exception:         # narrow this to timeouts and server errors
                    if attempt == attempts - 1:
                        raise             # give up; the next run starts from this page
                    time.sleep(2 ** attempt)
            f.write(json.dumps({"page": i, "answer": answer}) + "\n")
            f.flush()                     # on disk before the next paid call

call_model is your own function: it sends one page to the model and returns the answer as text. Each line of results.jsonl is one finished page, so a crash costs at most the call in progress, and the next run carries on from there. The job now costs N ÷ p calls at any length: 105 for the PDF at 95%.

Why not catch the error and skip the page? The job would always finish and never pay twice, but at 95% about 5 pages in 100 would be quietly missing, and the summary would have holes you can't see.

What this doesn't solve

A page too long for the model fails on every attempt: let the job stop, fix the cause, and the checkpoint keeps the rest. The math treats failures as independent, but in an outage they come together; that's why the snippet waits longer each time and then gives up. Retry only idempotent steps, ones that are safe to run twice: a call that also emails a customer isn't. And no gateway can do this for you, including the one I work on: it sees single calls, not your job, so it can't know which 70 pages you already have.

The takeaway

A job of N calls finishes with probability p to the power N. Make one call, not the whole job, the thing you retry, and save each answer before you pay for the next.