如何在 Python 中構建具備速率限制感知能力的 LLM 路由器：重試、退避與容量評分

Building a Rate-Limit-Aware LLM Router in Python: Retries, Backoff & Capacity Scoring

Most "LLM router" projects treat rate limits as an afterthought. They don't. This post walks through the pieces that make a router actually survive a provider's 429s in production.

If you've ever had your app die because OpenAI hit a 429 Too Many Requests mid-request and your only fallback was "try again in 1 second in a tight loop," you know the pain. That pattern burns you twice: you hammer the provider faster, and you have no idea how long to actually wait.

The fix isn't a longer sleep. It's reading the provider's own headers and building a router that's capacity-aware. That's the core engineering problem this post tackles, using SentioGate as the reference implementation.

The three pieces you need

A rate-limit-aware LLM router is three things glued together:

1. Header decoding — parse Retry-After, x-ratelimit-limit-*, x-ratelimit-remaining-* from each response
2. Capacity scoring — turn those numbers into a "how full is this provider right now" score that feeds the router's candidate ranking
3. Backoff-aware retry — when you do have to retry, wait the server-specified Retry-After, not a blind 1-second loop

Each one is small. Together they're the difference between "occasionally 429s" and "never surfaces a 429 to the user."

1. Decode the headers

Providers are inconsistent. OpenAI sends x-ratelimit-limit-tokens, x-ratelimit-remaining-tokens, retry-after. Anthropic sends anthropic-ratelimit-requests-remaining, anthropic-ratelimit-requests-reset. Gemini sends retry-after + its own quota headers. Groq, Mistral, DeepSeek — each has its own flavor.

Your router's provider adapter layer normalizes them:

@dataclass
class CapacityState:
    remaining_requests: int      # requests left in current window
    reset_at: datetime           # when the window refreshes
    remaining_tokens: int | None # if the provider reports token quotas
    reset_tokens_at: datetime | None
    is_rate_limited: bool        # last response was a 429

Each provider adapter's job is to fill in one of these from its response. The router core never knows which provider it's talking to — it only sees CapacityState.

2. Score the capacity

The router's scoring layer already ranks candidates on cost, latency, reliability, quality, and — crucially — capacity. The capacity score is derived from CapacityState:

• High remaining, soon reset → high score (go here)
• Low remaining, far reset → low score (avoid, or wait)
• is_rate_limited = True → circuit breaker: score drops to near-zero for the duration of the Retry-After window, then the router re-probes once to see if the window cleared

This is where most naive implementations go wrong. They treat a 429 as "retry in 1s" instead of "this provider is *unavailable until time T, rank it last until then." The circuit-breaker framing is the difference between a router that survives and one that just shuffles the same failing candidate around.

3. Backoff-aware retry

When the router must retry (you're out of alternatives, or the failure was transient), it uses the server's Retry-After as the base wait:

def next_wait(retry_after_s: int, attempt: int, backoff_base: float = 0.5):
    # Respect the server; floor at the exponential backoff so you don't
    # trust a 0 from a flaky provider.
    server_wait = max(retry_after_s, 0)
    backoff_wait = backoff_base * (2 ** (attempt - 1))
    return max(server_wait, backoff_wait)

Key detail: max(), not +. If the server says "retry in 30s" and your backoff says "wait 2s," you wait 30s. If the server says "retry in 0" (a bug or a 500, not a 429), your backoff is the floor. You never blind-loop.

The router caps total retries (max_retries) and enforces a wall-clock deadline on the whole request, so a pathological provider can't hang your app.

4. The wall-clock deadline

This is the part people forget. Retries + backoffs add up. A request that's "30s timeout, 2 retries, 0.5s base backoff" can actually run for over 70s before giving up. Your front-end is long gone.

So the router carries a single wall-clock deadline for the entire logical request (all retries included). Each candidate attempt gets min(candidate_timeout, time_until_deadline). When the deadline passes, the router stops and returns what it has (or a structured error) — it never runs past the deadline.

This is what "hardened transport" means in the intro post.

5. Capacity-aware failover (the payoff)

With all three pieces in, the router's behavior under pressure is:

1. Request comes in for model: "auto"
2. Router scores all candidate providers (cost, latency, reliability, quality, capacity)
3. The most-healthy provider is picked — not the one with the lowest cost, the one with the best combined score right now
4. If that provider 429s, the circuit breaker drops its capacity score to zero for the Retry-After window
5. The request routes to the next-best candidate — seamlessly, in the same logical request
6. The user sees one clean response, never the 429

That last line is the goal: the user never knows a provider failed. The router absorbs it.

Minimal reference implementation

Here's the skeleton (abridged) of the core routing loop you'd build around these pieces:

def route(request, providers, deadline):
    scored = rank(providers, request)          # cost+latency+reliability+quality+capacity
    for attempt, provider in enumerate(scored):
        if time.time() > deadline:
            return DeadlineExceeded()
        if provider.capacity.is_rate_limited:
            continue                            # circuit breaker: skip for now
        try:
            return provider.invoke(request, timeout=min(provider.timeout, deadline - time.time()))
        except TransientError as e:
            update_capacity(provider, e.headers)  # decode Retry-After / x-ratelimit
            if attempt + 1 >= max_retries:
                raise
            time.sleep(next_wait(e.retry_after_s, attempt + 1))
    return NoProviderAvailable()

The whole system is ~200 lines once the provider adapters exist. The adapters are the real work — and they're the part SentioGate's SDKs handle for you.

Why not just use httpx retries?

httpx / requests retry middleware is connection-level — it retries the same URL with the same body. It has no idea that a 429 means "this provider is capped, rank another one." A router is decision-level — it picks a different provider on the next attempt. That's the entire value proposition, and it's why a plain retry loop isn't enough.

Links

• SentioGate — the BYOK LLM router
• Python SDK · JS/TS SDK
• Live provider benchmarks (latency, TTFT, 429 rate)

This is the engineering post of the series. The companion posts: intro, semantic compaction, and custom providers. Tell me below what I should cover next.