Bifrost v2.3.0 的 TTFT 截止時間如何處理停滯串流並切換備援

TL;DR

• Time to first token (TTFT) is the wait between sending a streaming request and receiving its first output token, and it is the latency users feel most in chat and coding tools.
• A provider that accepts a stream and then stalls never returns an error, so ordinary retries and fallbacks do not fire until the full request timeout runs out.
• Bifrost v2.3.0 adds ttft_timeout_ms to routing rules: an attempt with no first token by the deadline is closed and the next fallback runs, and the client never sees the slow attempt.
• The last attempt in the chain is never cut off, a TTFT miss does not retry the same provider, and every cutoff is recorded in the attempt trail as ttft_timeout.

Time to first token (TTFT) is the time from sending a streaming request to receiving the first token of the response. Bifrost, the open-source AI gateway built in Go, can now enforce a TTFT deadline per routing rule: when a provider accepts a stream but produces no output in time, Bifrost closes that attempt and moves to the next fallback before anything reaches the client. This tutorial configures the cutoff, chooses a value from measured TTFT, and shows how to confirm which provider served each stream.

What time to first token measures

TTFT covers everything between the request leaving the client and the first piece of real output arriving: network time, provider queueing, prompt processing (prefill), and the generation of the first token. For a streaming chat or coding assistant, it is the moment the user stops staring at a spinner, which is why it matters more to perceived speed than total generation time.

Streams are typically delivered over server-sent events, so a provider can open the connection, send headers and startup events, and still produce no text. Bifrost counts only real output as the first token: text, reasoning, tool call arguments, audio or image data, a finish reason, or usage. Startup events such as response.created or an empty assistant role delta do not count, as described in the routing rules documentation.

For the techniques that lower TTFT itself (prompt size, caching, model choice, region), read the guide to cutting time to first token. This tutorial covers the other half: what to do when one provider's TTFT spikes and you want the request served somewhere else.

Why a stalled stream is worse than a failed request

A provider that fails fast is easy to handle. It returns a 429 or a 5xx, and the gateway retries or falls back to another provider. A provider that accepts the request and then stalls is harder, because nothing has failed yet. The connection is open, no error has arrived, and the request will only end when the request timeout expires, which is commonly set to 30 or 60 seconds so that long generations can finish.

That gap creates three problems:

• Users wait for the full timeout. A stall that lasts until the timeout turns a two-second answer into a minute-long one, and most users abandon the session before then.
• Fallbacks start late. The fallback provider only gets the request after the timeout, so even a successful fallback arrives far past any useful latency budget.
• Raising the timeout does not help. A shorter request timeout would cut off healthy long generations along with the stalled ones.

What you need is a separate deadline for the first token only, short enough to catch a stall and independent of how long the full generation may take. The automatic fallback walkthrough covers fallbacks for outright failures; a TTFT cutoff extends the same chain to slow starts.

How the Bifrost TTFT cutoff works

You set ttft_timeout_ms on a routing rule. When that rule matches a streaming request, each attempt that still has a fallback after it gets the deadline. If an attempt has not produced its first token in time, Bifrost closes its connection and moves to the next fallback. Because nothing is streamed to the client until the first token arrives, the client never sees the abandoned attempt.

Figure 1: Only attempts with a fallback after them get the deadline, and the client sees nothing until a first token arrives.

The rules that govern the cutoff:

Figure 2: Any no answers the attempt to the normal request timeout instead of the TTFT deadline.

The cutoff lives on routing rules because rules already decide the target and fallback chain for a class of traffic. That lets you give a latency-sensitive chat route a tight deadline while batch or agent traffic on the same LLM gateway keeps the default behavior.

Step 1: Configure providers and retries

Start with a running gateway and at least two providers that can serve the same model family. The gateway setup guide gets Bifrost running with npx or Docker:

npx -y @maximhq/bifrost

Add the providers in the Web UI at http://localhost:8080 or in config.json. Each provider has its own network_config, which sets retries for errors and the request timeout that the last attempt in a chain still runs under:

{
  "providers": {
    "openai": {
      "keys": [{ "name": "openai-1", "value": "env.OPENAI_API_KEY", "models": ["*"], "weight": 1.0 }],
      "network_config": {
        "max_retries": 2,
        "retry_backoff_initial": 500,
        "retry_backoff_max": 5000,
        "default_request_timeout_in_seconds": 60
      }
    },
    "azure": {
      "keys": [{ "name": "azure-1", "value": "env.AZURE_API_KEY", "models": ["*"], "weight": 1.0 }]
    }
  }
}

The defaults are max_retries: 0, a 500 ms initial backoff, and a 5000 ms cap, as listed in the retries reference. The rule in Step 2 also falls back to Bedrock, so configure it the same way. Azure, Bedrock, and Vertex AI need extra key fields such as deployments or regions; the provider configuration guide lists them. Retries still matter with a TTFT cutoff in place: a 429 or 503 on the primary is retried according to max_retries before any fallback runs, while a TTFT miss skips straight to the fallback.

Step 2: Add a routing rule with ttft_timeout_ms

Routing rules are CEL expressions with a target, fallbacks, and now an optional TTFT deadline. This rule matches every request for gpt-4o, sends it to OpenAI, and falls back to Azure and then Bedrock, cutting off any attempt with a fallback still ahead of it after 1.5 seconds without a first token:

{
  "governance": {
    "routing_rules": [
      {
        "id": "fast-gpt",
        "name": "Fast GPT",
        "enabled": true,
        "cel_expression": "model == 'gpt-4o'",
        "targets": [{ "provider": "openai", "model": "gpt-4o", "weight": 1 }],
        "fallbacks": ["azure/gpt-4o", "bedrock/openai.gpt-oss-120b-1:0"],
        "ttft_timeout_ms": 1500,
        "scope": "global",
        "priority": 0
      }
    ]
  }
}

To add the deadline to an existing rule without editing config.json, update it through the API:

curl -X PUT http://localhost:8080/api/routing/rules/fast-gpt \
  -H "Content-Type: application/json" \
  -d '{"ttft_timeout_ms": 1500}'

In the dashboard, the same field appears as Time to first token cutoff (ms) on the routing rule form. The expression language is CEL, so you can narrow the deadline to a tenant, a header, or a team. For example, model == 'gpt-4o' && headers["x-surface"] == "chat" limits the cutoff to traffic your chat front end tags, while background jobs that call the same model keep the normal timeout. The routing and fallback article explains how scopes and priorities decide which rule wins.

Applications do not change. Any client that already calls Bifrost through the OpenAI or Anthropic SDK by changing the base URL, as in the drop-in replacement guide, gets the cutoff as soon as the rule matches:

curl -N http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "stream": true,
       "messages": [{"role": "user", "content": "Summarize the incident report in three bullets."}]}'

Step 3: Choose a cutoff value from your TTFT data

A cutoff that is too tight abandons healthy requests and moves traffic away from your preferred provider. One that is too loose never fires. Pick the value from measured TTFT rather than a guess.

Bifrost exports a bifrost_stream_first_token_latency_seconds histogram through its Prometheus integration, labeled with provider, model, routing_rule_name, and fallback_index. This query returns the 95th and 99th percentile TTFT per provider and model over the last day:

histogram_quantile(0.99,
  sum by (le, provider, model) (
    rate(bifrost_stream_first_token_latency_seconds_bucket[1d])
  )
)

Change 0.99 to 0.95 for the p95 line. A practical starting point is a cutoff somewhat above the primary provider's normal p99 TTFT, so healthy requests almost never trip it and real stalls do. Long prompts raise TTFT because prefill takes longer, so a route that carries large contexts needs a higher value than a short chat route; give those routes separate rules rather than one compromise number.

Reasoning output counts as a first token for the cutoff, so a model that streams its reasoning is not cut off while it thinks. A model that reasons without streaming any of it can take far longer to emit output, so measure it before putting it behind a tight rule. The AI observability overview covers the other gateway metrics that sit beside this histogram.

Step 4: Verify which provider served the stream

After the rule is live, confirm the cutoff fires where you expect and that fallbacks serve the requests it cuts. Figure 3 shows the four places a cutoff is visible.

Figure 3: Headers answer one request, logs explain each attempt, and the histogram tells you where to set the cutoff.

Response headers. On a stream, Bifrost writes its headers before the first chunk. When a fallback served the request, the response carries x-bifrost-routing-info-is-fallback: true, the primary's provider and model in x-bifrost-routing-info-primary-provider and -primary-model, and x-bifrost-fallback-index with the fallback's position counting from 1. The response headers reference lists every header:

curl -sN -D - -o /dev/null http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "stream": true, "messages": [{"role": "user", "content": "ping"}]}' \
  | grep -i '^x-bifrost-'

Request logs. Each request's log entry includes an attempt_trail. A cut attempt appears there with ttft_timeout, alongside any key rotations or retries that happened before it. The built-in observability documentation shows where to find the trail in the logs view.

Routing engine log. Each cutoff writes a line in the form TTFT timeout: <provider>/<model> produced no first token within <duration>, which is useful when tailing a single gateway during a test.

Prometheus. Filter the first-token histogram by fallback_index to see how much streaming traffic the fallbacks absorb. A sudden rise in fallback-served streams for one rule usually means the primary's TTFT moved, which is worth an alert. The governance and observability post shows how these metrics feed dashboards.

How a TTFT cutoff interacts with retries and failure classes

Bifrost classifies every failed attempt with a failure_class and records it on the attempt trail. A stream cut by the TTFT deadline belongs to the timeout class, keeps ttft_timeout as its reason, and is recorded with status 504. The failure classification table shows how each class drives retries and key rotation:

Two consequences follow. First, a TTFT miss and a provider error take different paths: the error spends the provider's retry budget, while the TTFT miss goes straight to the fallback because retrying a slow provider adds more latency. Second, each fallback runs with its own full retry budget, so a fallback that returns a 429 still rotates keys before the chain moves on.

Load balancing across keys and providers works alongside the cutoff. If you weight a rule's targets across OpenAI and Azure, Bifrost picks one per request and applies the deadline to whichever it picked. In Bifrost Enterprise, adaptive routing shifts traffic away from degraded keys and providers over time, while the TTFT cutoff rescues the individual request that hit the slow one. The adaptive routing deep dive explains what that layer observes.

One cost to plan for: a provider may still bill the prompt tokens of an attempt that was cut off, because it already received the request. A tight cutoff on a route with very long prompts can therefore pay for the same prompt twice whenever it fires.

Common TTFT cutoff mistakes

• Setting a cutoff on a rule with no fallbacks. Nothing happens: the only attempt is also the last attempt, and the last attempt is never cut off. Add at least one fallback.
• Expecting it on non-streaming calls. The deadline applies only to streams. For non-streaming calls, tune the request timeout in network_config instead.
• Using one value for every route. Long-context and reasoning traffic has a higher normal TTFT. Separate rules per route keep the deadline meaningful on each.
• Making the fallback slower than the primary. The fallback should be a provider with comparable or better TTFT for that model. A reliable fallback design orders fallbacks by availability and latency, not only by price.
• Ignoring the bill. Cut attempts can still be billed for prompt tokens. Watch fallback-served volume on the cost side of your gateway before tightening the value.

Frequently asked questions

What is time to first token (TTFT)?

TTFT is the time between sending a streaming request to a language model and receiving the first token of its response. It includes network time, queueing at the provider, prompt processing, and generation of the first token, and it is the main driver of how fast a chat interface feels. The TTFT reduction guide covers ways to lower it.

What is a good TTFT for an LLM application?

It depends on the model, prompt length, and provider, so measure it rather than adopt a fixed target. Compare your own p95 and p99 TTFT per provider and model from the bifrost_stream_first_token_latency_seconds histogram, and set expectations per route, since long-context and reasoning requests have a higher normal TTFT than short chat turns.

Does a TTFT cutoff work for non-streaming requests?

No. ttft_timeout_ms applies only to streaming requests. Non-streaming requests ignore it and run under the provider's request timeout, configured as default_request_timeout_in_seconds in the provider's network settings, as shown in the provider network settings.

Will the client see a partial response from the cut provider?

No. Bifrost streams nothing to the client until an attempt produces its first token, so an attempt that is cut off before that point is invisible to the caller. The stream the client receives comes entirely from the attempt that answered, and the response headers say whether that was a fallback.

Does Bifrost retry the same provider after a TTFT timeout?

No. A TTFT miss goes straight to the next fallback without using the provider's max_retries, because retrying a provider that is slow to start usually adds latency. Errors such as 429 and 503 still use the retry budget before a fallback runs, as described in the retries and fallbacks documentation.

Am I billed for an attempt that was cut off?

You may be. The provider already received the request, so it can bill the prompt tokens even though Bifrost closed the connection before any output. Keep the cutoff above the primary's normal TTFT so it fires only on genuine stalls, and track fallback-served volume in the Bifrost LLM gateway metrics.

Keep streaming responses fast with Bifrost

A stalled stream costs users the full request timeout unless something cuts it short. A TTFT cutoff on the routing rule turns that stall into a quick, invisible failover, while retries and the last attempt keep their normal behavior. To try TTFT cutoffs, fallbacks, and routing rules on your own traffic, book a demo with the Bifrost team, or compare gateway latency in the Bifrost benchmarks.