如何在更改聊天機械人提示詞前評估 RAG 檢索效果

Before tuning a prompt, check if the retriever actually surfaced the evidence. A fluent answer often hides a missing document. And a retrieved document can still produce a wrong answer. Keep those failures separate when evaluating a retrieval-augmented generation pipeline.

This guide adapts AI Automation Blueprint's chatbot-accuracy topic into an offline retrieval test. It relies on document IDs and hand-labeled relevance judgments. It makes zero calls to an LLM, connects to no vector database, and promises no automatic accuracy bump.

What should a retrieval test contain?

Pull questions directly from the tasks the chatbot must handle. For each, log which documents carry enough evidence for an answer. Track the document revision alongside the label. A return-policy test breaks fast if someone updates the underlying policy file mid-run.

Use synthetic questions or appropriately approved examples. Customer conversations may contain personal information, and pasting them into a test fixture does not remove it. A working test suite can start tiny. But include awkward phrasing, not just questions that mirror a section heading.

Pinecone's RAG evaluation guide separates retrieval metrics like precision and recall from generation quality. Those scores measure retrieved evidence. They do not certify the generated answer.

How can you score a saved result locally?

The Python snippet below counts relevant document IDs within a fixed window. It rejects duplicate IDs so repeated records cannot pad the score. When the expected set is empty, it returns an unscored status. Questions outside the corpus require an explicit abstention check, not an invented recall figure.

def score_retrieval(retrieved, expected, k):
    if not isinstance(k, int) or isinstance(k, bool) or k <= 0:
        raise ValueError("k must be a positive integer")
    top = retrieved[:k]
    if len(top) != len(set(top)):
        raise ValueError("duplicate document IDs in result window")
    relevant = set(expected)
    if not relevant:
        return {"status": "unscored_no_relevance_labels"}
    hits = len(set(top) & relevant)
    return {
        "status": "scored",
        "precision_at_k": hits / k,
        "recall_at_k": hits / len(relevant),
    }

Precision here divides by the requested window size. If a system returns fewer documents than requested, empty slots count as misses. State that convention clearly before comparing runs against another team's harness. The function expects hashable IDs and validated inputs. It won't sanitize bad ingestion data.

This code has been checked against local fixtures only. It has not been integrated with Pinecone or a live chatbot here. Treat it as a scoring sanity check, nothing more.

How do you run a useful comparison?

1. Freeze the question set, relevance labels, and document revisions.
2. Save retrieved IDs for the current configuration, preserving rank.
3. Change one retrieval setting, then rerun the identical question set.
4. Compare per-query shifts alongside aggregate scores before keeping the change.

Never massage test labels to salvage a favorite parameter tweak. If a label is wrong, update it in source control and re-benchmark both runs. Keep unanswered questions in the numbers. Dropping them makes a broken index look passable.

Pinecone's RAG overview also recommends pairing questions with reference answers. Run an isolated generation review to confirm claims match retrieved snippets, citations reference the current revision, and missing evidence triggers a clean refusal.

Does a higher retrieval score prove better answers?

No. A retriever can fetch the right policy while the model misquotes an exception. Test the generation layer on its own. A convincing answer is not proof of good retrieval, either. Bind the test question, returned IDs, and output together so you can isolate the break without guessing.

Expand the test set whenever an unhandled customer query slips past. Keep every older test in place so today's patch doesn't silently break yesterday's fix.