LLM 應用的語義快取:何時安全重用答案,何時會出錯
Semantic caching for LLM apps: when reusing an answer is safe, and when it's a bug
Ali Ahmed 介紹 LLM 應用的語義快取,説明系統如何按問題嵌入向量的相似度重用已儲存答案,而毋須呼叫模型。測試示例中,「Cancel my order」和「Don't cancel my order」的相似度為 0.866;文章建議以標註配對調校高閾值,並按模型、系統提示詞及租戶隔離快取。個人或帳戶相關、依賴工具或多輪對話,以及高風險問題,不宜使用語義快取。
A semantic cache stores LLM answers with an embedding of the question, and when a new question's embedding is similar enough to a cached one, it returns the stored answer instead of calling the model. A hit costs a few milliseconds and no tokens. The danger is in "similar enough": embeddings capture topic, not meaning precisely, so "cancel my order" and "don't cancel my order" can look almost identical. A safe semantic cache uses a high threshold, scopes every entry to the model, system prompt and tenant, expires entries, and is only used for questions whose answer doesn't depend on who's asking. Here's a tested Python implementation and the failure cases.
How does a semantic cache work?
- Embed the incoming question.
- Compare it with the embeddings of cached questions in the same scope (cosine similarity).
- If the best match is above the threshold, return its answer. Otherwise call the model and store the result.
An exact-match cache (hash of the prompt) is the safe baseline: it only hits on identical text. A semantic cache trades some of that safety for a higher hit rate on rephrasings.
```python title="semcache.py"
import hashlib
import time
import numpy as np
class SemanticCache:
"""Reuse an LLM answer when a new question is close enough to a cached one, within the same scope."""
def init(self, embed, threshold: float = 0.95, ttl_seconds: float = 3600, clock=time.time):
self.embed, self.threshold, self.ttl, self.clock = embed, threshold, ttl_seconds, clock
self.entries: dict[str, list[tuple[np.ndarray, str, str, float]]] = {}
@staticmethod
def scope(model: str, system_prompt: str, tenant: str) -> str:
# A different model, instructions or customer means a different answer: never share across scopes.
return hashlib.sha256(f"{model}\0{system_prompt}\0{tenant}".encode()).hexdigest()
def get(self, scope: str, question: str):
q = self.embed(question)
now = self.clock()
live = [e for e in self.entries.get(scope, []) if now - e[3] < self.ttl]
self.entries[scope] = live
if not live:
return None
sims = np.array([v @ q for v, *_ in live])
best = int(sims.argmax())
return (live[best][2], float(sims[best]), live[best][1]) if sims[best] >= self.threshold else None
def put(self, scope: str, question: str, answer: str):
self.entries.setdefault(scope, []).append((self.embed(question), question, answer, self.clock()))
`embed` must return unit-length vectors, so the dot product is the cosine similarity. A hit returns the answer, the similarity and the cached question: **log all three**, because they're what you'll need when a user reports a wrong answer. At scale, the linear scan becomes a vector index; see [embeddings and vector search, explained](https://aliahmed.si/blog/embeddings-vector-search-explained/).
## Where does it go wrong?
To keep the tests self-contained I used a stand-in embedding (hashed bag of words). Real embedding models score these pairs differently, but the pattern is the same: wording that changes the meaning barely changes the vector.
| Cached question | New question | Similarity |
| --------------------------- | ------------------------------------------------ | ---------- |
| What is your refund policy? | what is your refund policy | 1.000 |
| Cancel my order | Don't cancel my order | 0.866 |
| What is your refund policy? | What is your refund policy for orders over $100? | 0.745 |
| What is your refund policy? | How do I reset my password? | 0.000 |
"Don't cancel my order" sits at 0.866, closer than many genuine rephrasings would be. With real models, negations and small qualifiers often score above 0.9. That's why:
* **The threshold is high** (0.95 here), and you tune it on **labelled pairs**: rephrasings that must hit, and near-misses that must not. Measure false hits the way you'd measure [face match false accepts](https://aliahmed.si/blog/face-match-threshold-far-frr/), because a wrong cached answer is a false accept.
* **The scope is part of the key.** The tests confirm that the same question in `tenant-b` misses `tenant-a`'s entry. Changing the system prompt or model also starts a fresh scope, so a prompt fix takes effect immediately.
* **Entries expire.** A refund policy changes; a 60-second TTL in the test makes the cached answer disappear after a minute.
## When shouldn't you cache?
* **Personal or account-specific answers.** "What's my balance?" must never be served from another user's cache entry. If the answer depends on who's asking or on live data, don't cache it, or include the user in the scope.
* **Tool-calling turns.** If the model would call a tool, the answer depends on the tool result at that moment.
* **Multi-turn context.** The same final question means different things in different conversations. Cache only standalone questions, or include the conversation's state in the key.
* **Anything where a wrong answer is costly.** Compliance answers, prices, medical or legal questions: use an exact-match cache or none.
A good first use is an FAQ-style assistant answering general questions, with exact-match caching everywhere else. Provider-side **prompt caching** is a different thing: it reuses the processed prompt prefix to make calls cheaper and faster, but the model still runs. It's covered in [how to cut LLM API costs](https://aliahmed.si/blog/reduce-llm-api-costs/).
## Summary
* A semantic cache returns a stored answer when a new question's embedding is close to a cached one.
* Near-misses such as negations score high: use a **high threshold tuned on labelled pairs**.
* **Scope** entries by model, system prompt and tenant; **expire** them; **log** every hit.
* Don't cache personal, tool-dependent, multi-turn or high-stakes answers.
Related: [hybrid search with BM25 and reciprocal rank fusion](https://aliahmed.si/blog/hybrid-search-bm25-reciprocal-rank-fusion/) and [stopping an LLM from inventing numbers](https://aliahmed.si/blog/stop-llm-inventing-numbers/).
---
Written by Ali Ahmed · [Original article](https://aliahmed.si/blog/semantic-cache-llm-responses/)
來源:DEV Community · dev.to