AI 快取的實際效益：成本、速度、速率限制與可靠性

The Real Benefits of AI Caching: Cost, Speed, Limits, and Reliability

In Part 1 we described the four layers of AI caching: exact response caching, semantic caching, provider prompt caching, and embedding caching. This post answers the obvious follow-up: what do you actually get for turning them on?

The short answer is five things: lower cost, lower latency, more headroom under rate limits, better resilience, and more consistent answers. Below we go through each one, with worked examples you can adapt to your own traffic. We use ZRouter to show how each benefit shows up in practice.

A note on the numbers: provider prices change often, so the examples use round, illustrative figures. Plug in your own provider's current rates to get real estimates.

1. Lower cost

Response caching removes calls entirely

A response cache hit costs nothing at the model provider. Your savings are simply your hit rate multiplied by your spend on those requests.

Worked example. A customer support assistant handles 100,000 requests a month. Each request averages 1,500 input tokens and 300 output tokens. Assume a model priced at $2.50 per million input tokens and $10 per million output tokens.

• Input: 100,000 × 1,500 = 150M tokens → $375
• Output: 100,000 × 300 = 30M tokens → $300
• Total: $675 per month

Support traffic is highly repetitive. If the exact cache catches 15% of requests and the semantic cache catches another 20%, then 35% of calls never reach the provider:

• Model spend: 65% × $675 = $439
• Semantic lookups: about 85,000 embedding calls (every request that missed the exact cache) at roughly 20 tokens of final user text each. That is around 1.7M embedding tokens, which costs cents at typical embedding prices.

That is about $236 saved per month on one assistant, before counting the latency improvement.

Prompt caching shrinks the repeated part

For workloads where every answer must be fresh, like agents, coding tools, and long chats, response caching rarely hits. Prompt caching still helps, because most of each request is the same prefix.

Worked example. A coding agent session runs 40 turns. Each turn sends a 25,000-token stable prefix (system prompt, tool definitions, project files) plus a growing conversation. Ignore the conversation growth and focus on the prefix:

• Without prompt caching: 40 × 25,000 = 1M input tokens at full price.
• With prompt caching: the first turn pays full price (some providers add a small write premium). Turns 2 to 40 read 975,000 prefix tokens at the provider's cached rate.

Providers discount cached input heavily. Depending on provider and model, cached input tokens are billed at roughly 10% to 50% of the normal input price. At a 90% discount, the prefix cost falls by close to 85% across the session. At a 50% discount, it still falls by close to half.

The catch is that you only get this if the prefix is actually identical every turn and the request reaches the same provider, the same model, and often the same API key. That is harder than it sounds, and it is one of the main reasons to handle it at a gateway (see Part 4). ZRouter's prompt-cache planner adds the right directive for each provider automatically, and its session keeping feature pins every request in a detected session to the same target and the same provider key so the cache stays warm.

2. Lower latency

A model call has two slow parts: reading the prompt (time to first token) and writing the answer (generation time).

• Response cache hits skip both. A Redis lookup takes a few milliseconds. A typical chat completion takes one to ten seconds. For the user, a cached answer feels instant.
• Semantic cache hits add an embedding call and a vector search before the answer, usually tens of milliseconds, which is still far faster than generating.
• Prompt cache hits cut time to first token, because the provider does not reprocess the long prefix. On very long prompts this can be the difference between a two-second pause and a near-immediate start.

Latency matters beyond user patience. Agents chain many calls together. Saving a second on each of 40 steps saves 40 seconds on one task.

In ZRouter, streaming responses are cached as the raw event stream, so a cached streaming answer still arrives as a stream. Your chat UI renders it the same way, just much faster.

3. More headroom under rate limits

Providers limit requests per minute and tokens per minute. When you hit a limit, requests fail or queue.

Every response cache hit is a request that does not count against your provider limits. Every prompt cache hit can reduce the input tokens that count against token-per-minute limits on providers that discount cached tokens in their limit accounting.

A subtler benefit comes from request coalescing. Picture a product launch, where hundreds of users ask the same question in the same minute. A plain cache misses on all of them until the first answer returns. ZRouter coalesces concurrent identical misses in each gateway process: one request goes to the provider, and the rest wait and replay its answer. A spike of identical traffic becomes one provider call instead of hundreds.

4. Better resilience

A cache is also a buffer against provider trouble. During a brief outage or a period of heavy throttling, popular questions can still be answered from cache because the lookup happens before the request is sent anywhere.

ZRouter is careful about one edge case here. If a request is answered by a failover target rather than the primary target, the response is not cached. Otherwise the backup model's answer would be stored under the primary model's key and served long after the primary recovered. Entries stored before the outage keep serving until they expire.

5. More consistent answers

Language models do not always give the same answer to the same question. For some products, that variation is a problem: a pricing FAQ, a policy explanation, or a compliance answer should not change from one user to the next.

A response cache makes repeated questions return the same reviewed answer until the entry expires. That makes support quality easier to audit and reduces "the bot told me something different yesterday" tickets.

What caching does not buy you

To keep expectations honest:

• It does not make novel requests cheaper. A brand new question with a brand new context gets full price, except for any cached prefix.
• It does not improve answer quality. A cache replays answers. A bad answer cached is a bad answer served many times, which is why expiry (TTL) matters.
• Semantic caching is not free. Each miss still pays for an embedding and a vector search. If your traffic is almost never repetitive, those lookups are pure overhead.

Measuring the benefit

Turning a cache on is easy. Knowing whether it pays off takes data. Track at least:

• Hit rate per layer (exact, semantic, prompt).
• Cost avoided: provider spend you would have paid without the hits.
• Latency for hits versus misses.
• Wrong-hit reports for the semantic layer, which tell you if your threshold is too loose.

ZRouter records each cache hit as a usage entry tagged cache_type: exact or cache_type: semantic, with the caller's user path. By default, usage totals only count provider-bound requests so your cost reports match your provider bill. Add cache_mode=all to include hits, or cache_mode=cached to see only hits:

# Provider spend only (default)
curl "https://your-gateway/admin/usage/summary" \
  -H "Authorization: Bearer $AIGATEWAY_MASTER_KEY"

# Only cache hits
curl "https://your-gateway/admin/usage/summary?cache_mode=cached" \
  -H "Authorization: Bearer $AIGATEWAY_MASTER_KEY"

There is also a cache overview at /admin/cache/overview, and every cached response carries an X-Cache header you can log on the client side.

Key takeaways

• Response caching saves the entire call and is the biggest win for repetitive traffic.
• Prompt caching is the biggest win for long, fresh, multi-turn work like agents and coding tools.
• Caching also buys latency, rate-limit headroom, resilience, and consistency.
• Measure hit rates and cost avoided per layer, or you are guessing.

Next in the series: [Part 3: When to Use AI Caching, and When Not To]