nRouter 分享 LLM 呼叫的重試與停止重試規則
What's your rule for when an LLM call should NOT be retried?
nRouter 分享 LLM 呼叫的重試規則：對逾時、429 和 5xx 採用退避及 jitter，並以伺服器的 Retry-After 為準；400/401/403 則不重試，應立即回報並修正請求。重試應受任務總預算限制；若生成在串流中途停止，應重新開始該訊息，不要接續半成品。對高峯期服務商回傳的 529 或過載錯誤，作者尚未決定應原位重試，還是立即切換模型並接受品質變化。
Posted on Oct 27 AI-assisted
Everyone has a retry story. Mine: an agent that retried a malformed request eleven times before anyone noticed, each attempt replaying the full conversation. The request was never going to succeed — a 400 doesn't heal — but the retry policy didn't care, because it only looked at "did it throw?"
Our working rules now:
- Retry timeouts, 429s and 5xx — with backoff, jitter, and the server's
Retry-Afterwinning over our schedule. - Never retry 400/401/403. Fail fast, surface it, fix the request.
- A retry budget, not just a count: retries get a slice of the task's total budget, then the task fails honestly.
- If it dies mid-stream, restart the message cleanly — never append to a half-generated answer.
What I haven't settled: provider 529s / overloaded errors during peak. Retry in place, or fail over to a different model immediately and accept the quality change?
What's your rule set — and which error taught it to you?
來源：DEV Community · dev.to