RAG 答案出錯？先檢查檢索，再調整提示詞

Someone asks your docs assistant how to rotate an API key. It answers confidently, cites a page, and the steps are from the old dashboard. The first instinct is to open the prompt and add another line: "Only answer from the provided context. Be accurate."

Sometimes that's the fix. Often it isn't, because the model may never have seen the right page in the first place. A retrieval-augmented generation (RAG) system is a pipeline, and a wrong answer at the end can start at any stage before it. If you don't check the stages in order, you end up tuning the prompt for a problem that lives in your chunking.

This article gives you a debug order for bad RAG answers: what to check first, how to tell retrieval failures from generation failures, and the small test that makes the difference visible.

The pipeline, in one paragraph

Most RAG systems follow the same five steps. You chunk your documents into pieces, embed each piece as a vector, retrieve the pieces closest to the question, optionally rerank them with a more careful model, and generate an answer from the top results. The original RAG paper (Lewis et al., 2020) combined a retriever with a generator in exactly this spirit: fetch relevant text at question time instead of relying only on what the model memorized.

The useful consequence: each step fails in its own way, and each can be checked on its own. That's what makes a bad answer diagnosable instead of a mystery.

Step 0: Separate the two questions

Before changing anything, split "the answer was wrong" into two questions:

Did the right chunk reach the model? (retrieval)
Given the right chunk, did the model use it correctly? (generation)

To answer them, log what was retrieved for the failing question: the chunk IDs, their text, and their scores, in the order they were passed to the model. If your system doesn't log this yet, add it before anything else. Without it, every fix is a guess.

Then read the retrieved chunks yourself. If the correct information isn't there, stop looking at the prompt. You have a retrieval problem.

Step 1: Is the answer in the index at all?

Start at the beginning of the pipeline. Search your chunk store for a phrase you know is in the right document.

Common findings at this step:

The document was never ingested, or an old version is still indexed next to the new one.
The parser dropped content: tables flattened into noise, text in images skipped, a PDF extracted with columns interleaved.
The relevant sentence exists but sits in a chunk whose text no longer makes sense on its own.

That last one is the subtle one. A chunk that says "Click Rotate, then confirm" means nothing if the heading that said which key and which screen ended up in the previous chunk. Anthropic's write-up on Contextual Retrieval is built around this exact problem: individual chunks lose the context of the document they came from. Their fix is to prepend a short, generated description of where each chunk sits in its document before embedding and indexing it.

A cheaper first step is structure-aware chunking: split on headings and sections instead of a fixed character count, and keep the section title with the chunk.

Step 2: Is it retrieved, and how far down?

If the right chunk exists, check whether retrieval finds it. Run the failing question and look at where the right chunk ranks. Rank 1, rank 14, or not in the top 50 are three different problems.

Not found at all, but the question uses an exact term (an error code, a function name, a product SKU): pure vector search can miss exact matches. Anthropic's post gives the example of a query for "Error code TS-999", where an embedding model may return content about error codes in general and miss the exact match. Keyword search such as BM25 catches those, which is why many systems run hybrid search: keyword and vector retrieval together, merged into one list.

Found, but ranked too low to be passed to the model: this is where retrieving wide and reranking narrow helps. Pull more candidates than you need, then let a reranker put the best few on top. In Anthropic's tests, combining contextual embeddings with contextual BM25 reduced the top-20 retrieval failure rate by 49% (from 5.7% to 2.9%), and adding a reranking step brought the reduction to 67% (1.9%). Those are their numbers on their datasets, so treat them as a reason to test the pattern on yours, not as a promise.

Found in the top few, but next to near-duplicates: if three versions of the same page are indexed, the outdated ones compete with the current one. Deduplicate, or attach a version or date to each chunk and filter on it.

Step 3: Did the model use what it got?

Only now look at generation. If the right chunk was in the context and the answer is still wrong, the usual causes are:

Conflicting chunks. The old and new procedures were both retrieved, and the model picked the wrong one. Fix this upstream (Step 2) rather than asking the model to guess which is current.
Too much context. The right chunk is there, buried among many weak ones. Passing fewer, better-ranked chunks can help, but check your own data: Anthropic found passing 20 chunks worked better than 10 or 5 in their tests.
A prompt that doesn't ground. The model answers from general knowledge when the context is thin. Ask it to answer only from the provided sources, cite the chunk it used, and say so when the sources don't contain the answer.

Asking for citations has a second benefit: when an answer is wrong, the citation tells you which chunk misled it, which sends you straight back to the right step.

The 10-minute test that makes this repeatable

Debugging one question at a time doesn't tell you whether a change helped overall. A small retrieval test set does. Write down 10 to 20 real questions, and for each one note the chunk (or document) that contains the answer. Then measure how often that chunk shows up in the top k results.

# Each case: a real question and the ID of the chunk that answers it.
cases = [
    {"question": "How do I rotate an API key?", "expected": "docs/keys.md#rotate"},
    {"question": "What does error TS-999 mean?", "expected": "docs/errors.md#ts-999"},
    # ...add 10-20 real questions from your logs
]

def recall_at_k(search, cases, k=5):
    """search(question, k) must return a list of chunk IDs, best first."""
    hits = 0
    misses = []
    for case in cases:
        results = search(case["question"], k)
        if case["expected"] in results:
            hits += 1
        else:
            misses.append(case["question"])
    return hits / len(cases), misses

Run it before and after every change to chunking, embeddings, search or reranking. The score tells you whether retrieval got better; the list of misses tells you which questions to read by hand next. Keep generation quality as a separate check (did the answer match the source?), so a bad answer always points to one stage instead of the whole pipeline.

The debug order, as a checklist

Log retrieved chunk IDs, text and scores for every answer.
Confirm the answer exists in the index, in one current version, in a chunk that makes sense on its own.
Check where the right chunk ranks. Add keyword search for exact terms; retrieve wide and rerank narrow for ranking problems.
Remove conflicting and duplicate chunks before blaming the model.
Only then tune the prompt: ground it in sources, ask for citations, allow "not in the sources".
Re-run your retrieval test set after every change.

Where to go next

If you want the whole mental model first, the free RAG Quick-Start covers the five-step pipeline, the three decisions that move quality most (structure-aware chunking, hybrid search, retrieve wide and rerank narrow) and how to measure recall and faithfulness, in 6 pages you can read in about fifteen minutes.

When you're ready to go deeper, The Complete Guide to Retrieval-Augmented Generation spends 46 pages on every stage, evaluating retrieval and generation separately, debugging failures in pipeline order, agentic RAG, and a full worked example of a docs assistant built end to end.

This article was written with the assistance of AI and reviewed by the author.