RAG 全流程:各階段解決甚麼問題,失效時又會怎樣
RAG end to end: what each stage fixes, and what breaks when it fails
Deployed Lab 的文章逐一説明 RAG 的搜尋、權限檢查、重排序與生成階段各自修正的問題,以及失效時會造成的結果。文中的虛構分數示例顯示,混合語義與關鍵字搜尋可讓現行退款政策升至首位,答案為 30 天;只有權限篩選會移除僅限法律部門閲讀的頁面。文章提醒,重排序無法補回搜尋未找到的頁面,提示詞中的保密規則亦不能取代在組成提示詞前執行權限控制。
A support assistant tells a customer, "You have 14 days to request a refund." The real policy says 30 days. In the same demo it quotes a contract line that only Legal may read.
The English was fluent. The assistant was simply handed the wrong pages. This is the problem RAG (retrieval-augmented generation) exists to solve, and each stage of it fixes a different failure.
The example below is a composite with made-up scores. It shows a pattern, not a benchmark.
The idea: an open-book exam
An AI that has never seen your files will guess, and the guess sounds confident. RAG lets it look up the right pages just before it answers. Looking things up is cheap and always current. Retraining goes stale the day a policy changes.
Your documents are filed once, in advance, as chunks, each stamped with who may read it. Then every question passes through these stages.
The stages and how each one fails
1. Search. Hybrid search matches by meaning and by exact words, then mixes the scores.
- Failure: an exact code or name is not found. Meaning search is weak on strings like "V-7731".
- Fix: add keyword search.
2. Permission check. Pages the asker may not read are dropped, using the access list stored with each chunk.
- Failure: the asker sees a page they must not read.
- Fix: filter before the prompt is built. An AI cannot repeat what it never received.
3. Re-rank. A slower, more careful reader re-sorts the pages that search found.
- Failure: the right page is found but ranked too low to fit. Only a few pages fit in the prompt.
- Fix: add a re-ranker.
4. Writing. The rules, top pages and question go into one prompt, and the AI answers.
- Failure: a number appears that is in no page.
- Fix: tell it to answer only from the pages, name its sources, and say "I don't know".
One worked example
The question: "How long do Enterprise customers have to request a refund?" Four pages, made-up scores from 0 to 1:
| Page | Meaning | Keyword | Careful reader |
|---|---|---|---|
| Refund policy v3 (current) | 0.74 | 0.95 | 0.97 |
| Refund policy v1 (old) | 0.82 | 0.55 | 0.30 |
| Marketing FAQ | 0.88 | 0.35 | 0.20 |
| Legal-only contract page | 0.79 | 0.60 | 0.55 |
Only the top 3 reach the AI. With meaning alone, the FAQ wins and the current policy is cut. The answer is vague, and the Legal-only page is in the prompt.
Now mix half meaning and half keyword:
- Current policy: 0.5 x 0.74 + 0.5 x 0.95 = 0.845
- Old policy: 0.5 x 0.82 + 0.5 x 0.55 = 0.685
- Marketing FAQ: 0.5 x 0.88 + 0.5 x 0.35 = 0.615
The current policy rises to first place and the answer becomes 30 days. The re-ranker reaches the same result from a different stage: it scores the current policy 0.97 against 0.30 for the old one.
Only the permission filter removes the Legal-only page.
Which stage do you blame?
Check in order: shelving (chunk size), search, permission check, re-rank, writing. An early fault hides later ones. Read the pages that went into the prompt first, because a wrong number could be a stale page or the AI inventing it.
Caveats
- Real scores come from models, and real systems add steps, such as rewriting the question first.
- A re-ranker only sees the pages search returned. If search never found the right page, the re-ranker cannot save it.
- A rule in the prompt, such as "never reveal secrets", is a request and not a lock. Enforce access before the prompt is built.
Try it yourself
The chapter has an interactive lab where you switch hybrid search, the re-ranker and the permission filter on and off, and watch the answer change. It takes about 12 minutes and is free: RAG end to end. It is one of the free chapters of Deployed Lab, a course on AI engineering and the forward deployed engineer role.
來源:DEV Community · dev.to