模型呼叫逾時只代表本地停止等待：重試時的五個迷思

Your Timeout Is Local: Five Myths About Model Calls You Retry

My batch job died at 03:40. The box was up. The model was reachable. My retry logic was wrong.

I keep seeing the same five claims in code review. Each one sounds sensible. Each one quietly produces duplicate work, empty output, or a worker stuck forever.

So I stopped arguing and built a probe. One script, one verdict table, one ledger. No benchmarks here. Free-tier conditions change too fast for my numbers to help you.

Myth 1: "A timeout means the call did not happen"

Your deadline is measured on your machine. The request may already sit in a queue. Or it may already be done.

A local clock expiring tells you one thing only. You stopped listening. It says nothing about remote execution.

Corrected model: DEADLINE means unknown, not failed.

outcome = with_deadline(lambda: call_model(model, prompt), seconds=20)
if outcome.verdict is Verdict.DEADLINE:
    # remote state unknown: do not assume "no work done"
    log.warning("stopped listening fp=%s", fp)

This matters when the call has side effects. A tool call that writes a row, or sends a message, may have landed. Retry blind and you write twice.

Myth 2: "HTTP 200 means I got an answer"

A 200 says the transport worked. It does not say the text is usable.

I have seen three failure shapes arrive with a healthy status code:

1. Empty — whitespace only, or a blank field where text should be.
2. Truncated — ends mid-phrase because a cap was hit.
3. Off-target — fluent text that ignores the instruction.

Only the first two are cheap to detect. The third needs a check you wrote yourself, like a required field or a schema parse.

Corrected model: success has two gates. Transport success, then content success. Your retry policy should see both.

Myth 3: "Retries are free"

Every retry is another call. Another unit of time. Another chance to hit a limit.

Ask yourself a question. What is your worst case, attempts times deadline? If the answer is forty minutes, you do not have a retry policy. You have an outage.

Cap two numbers, not one:

• Attempts — a small integer, not a loop.
• Total wall clock — the budget the whole job may spend.

And treat THROTTLED differently from TRANSPORT. Backpressure wants a longer pause. A dropped connection wants a short jitter.

Myth 4: "Same prompt, same answer, so I can cache by prompt"

Outputs are snapshots, not constants. The same prompt can return different text on two runs.

That is fine for chat. It is a bug for a cache key.

Corrected model: the key is a fingerprint of everything that shapes the output.

def fingerprint(model: str, prompt: str, params: dict) -> str:
    blob = json.dumps({"m": model, "p": prompt, "params": params}, sort_keys=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16]

Model id, prompt, and parameters all belong in the hash. Swap the model and your cache must miss. Otherwise you serve yesterday's answer under a new label.

Myth 5: "Streaming means progress, so I can drop the timeout"

Streaming gives you bytes sooner. It gives you no guarantee of finishing.

A stalled stream is worse than a slow response. It holds a worker slot and looks alive. Your total timeout never fires because data keeps trickling.

Use two clocks:

• Idle timeout — no bytes for N seconds, give up.
• Total timeout — hard ceiling for the whole call.

If you only have one, you have the wrong one.

Build the probe

Here is the harness I actually run. It is provider-agnostic on purpose. Inject your client and keep your own numbers.

"""probe_call.py - classify model-call outcomes before you retry them."""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import threading
import time
from dataclasses import dataclass
from enum import Enum
from pathlib import Path

LEDGER = Path(os.environ.get("LEDGER", "calls.jsonl"))


class Verdict(str, Enum):
    OK = "ok"
    TRANSPORT = "transport"
    DEADLINE = "deadline"
    THROTTLED = "throttled"
    EMPTY = "empty"
    TRUNCATED = "truncated"


@dataclass
class Outcome:
    verdict: Verdict
    text: str = ""
    detail: str = ""
    seconds: float = 0.0


def fingerprint(model: str, prompt: str, params: dict) -> str:
    blob = json.dumps({"m": model, "p": prompt, "params": params}, sort_keys=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16]


def with_deadline(fn, seconds: float) -> Outcome:
    """Run fn in a thread. Return DEADLINE on overrun. Never block forever."""
    box: dict = {}

    def runner() -> None:
        try:
            box["text"] = fn()
        except Exception as exc:  # classify, do not swallow
            box["error"] = exc

    t = threading.Thread(target=runner, daemon=True)
    started = time.monotonic()
    t.start()
    t.join(seconds)
    elapsed = time.monotonic() - started

    if t.is_alive():
        return Outcome(Verdict.DEADLINE, detail="local clock expired", seconds=elapsed)
    if "error" in box:
        return Outcome(Verdict.TRANSPORT, detail=repr(box["error"]), seconds=elapsed)

    text = (box.get("text") or "").strip()
    if not text:
        return Outcome(Verdict.EMPTY, detail="status ok, no usable text", seconds=elapsed)
    if text.endswith((",", " and", " the")):
        return Outcome(Verdict.TRUNCATED, text=text, detail="ends mid-phrase", seconds=elapsed)
    return Outcome(Verdict.OK, text=text, seconds=elapsed)


def record(fp: str, out: Outcome) -> None:
    row = {"fp": fp, "verdict": out.verdict.value, "seconds": round(out.seconds, 3)}
    with LEDGER.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + "\n")


def seen(fp: str) -> str | None:
    if not LEDGER.exists():
        return None
    for line in LEDGER.read_text(encoding="utf-8").splitlines():
        row = json.loads(line)
        if row["fp"] == fp:
            return row["verdict"]
    return None


RETRYABLE = {Verdict.TRANSPORT, Verdict.THROTTLED, Verdict.DEADLINE}


def call_model(model: str, prompt: str) -> str:
    """Wire your client here. Return raw text. Raise on transport errors."""
    raise NotImplementedError("plug in your client, then rerun")


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--prompt", required=True)
    ap.add_argument("--model", default=os.environ.get("MODEL_ID", "unset"))
    ap.add_argument("--attempts", type=int, default=1)
    ap.add_argument("--deadline", type=float, default=20.0)
    args = ap.parse_args()

    fp = fingerprint(args.model, args.prompt, {})
    if seen(fp) == Verdict.OK.value:
        print(f"fp={fp} already OK; skipping on ledger hit")
        return

    for attempt in range(1, args.attempts + 1):
        out = with_deadline(lambda: call_model(args.model, args.prompt), args.deadline)
        record(fp, out)
        print(f"attempt={attempt} verdict={out.verdict.value} t={out.seconds:.2f}s")
        if out.verdict is Verdict.OK:
            print(out.text)
            return
        if out.verdict not in RETRYABLE:
            break
        time.sleep(min(2 ** attempt, 10) * 0.5)
    raise SystemExit(f"fp={fp} gave up; read {LEDGER}")


if __name__ == "__main__":
    main()

Two things make this useful beyond a toy. The ledger is append-only, so a restarted job can skip finished work. And the verdict is written before any retry decision.

Run the kill tests first

Do not test your retry path on a good day. Force the bad ones.

# 1) force the local-clock path
python probe_call.py --prompt "reply with: ok" --deadline 0.01

# 2) force empty and truncated shapes
#    point call_model at a stub that returns "" and "the"
python probe_call.py --prompt "reply with: ok" --attempts 1

# 3) force transport errors, then restore
#    drop your network briefly, or point at a closed port
python probe_call.py --prompt "reply with: ok" --attempts 3

# 4) prove the ledger gate
python probe_call.py --prompt "reply with: ok" --attempts 1

Step 4 is the one people skip. If you run the same fingerprint twice, you should see a skip, not a second call.

The decision table

The DEADLINE row is the one that needs judgement. Retrying the same fingerprint is safe when the call only reads. It is not safe when the call writes.

How I run this on a free host

I run the probe matrix on resources that cost me nothing, so a bad loop cannot turn into a bill. That is where MonkeyCode comes in for me: it offers free model access and a free server option, which is enough to run the kill tests above end to end.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server is exactly the kind of environment that exposes these myths. It can be slow, it can sleep, and it can be reclaimed. That is a feature for this experiment. It is not a latency baseline.

So read the verdicts. Ignore the wall-clock numbers. Rerun on your own host before you trust any of them.

What this harness does not prove

Be honest about the edges. Mine are sharp.

• It does not measure model quality. OK means text arrived, not that the text is right.
• It does not make calls idempotent. The ledger records attempts. Only your code can prevent double writes.
• The truncation check is a heuristic. Three suffix patterns is a demo, not a parser.
• It does not model your rate limits. Any specific attempt count I print would be a guess about your account.
• Threads are not cancellation. The DEADLINE branch abandons a thread, it does not kill the remote work.

That last one is the whole article in a sentence. You can stop waiting. You cannot un-ask.

Who should skip this

Some teams should not adopt this pattern. I would rather say so up front.

• Hard latency SLOs. A free host and a sleeping box make latency claims meaningless.
• Strict data residency. Read the terms of whatever host and model you pick. Do not take my word.
• High fan-out pipelines. A per-call thread plus ledger append gets clumsy fast. Use a real queue.
• Anything with money attached. If a duplicate write costs real money, build an idempotency key at the API layer first.

The one question to keep

Before you add a retry, ask it. If this call already ran, what breaks?

If you cannot answer, the retry is not the fix. The missing ledger is. Start with the probe, run the kill tests, and let your own verdicts shape the policy.

If you want a zero-cost place to run those kill tests, the free models and free server on MonkeyCode will do the job.