RAG 評估：重要的檢索與生成指標

Most RAG systems that silently decay in production decay in the retrieval layer. The LLM output still looks fluent, so the surface tests pass. Users just notice that the answers get subtly wrong — citing the wrong clause, mixing up two customers, giving an outdated version of the policy. By the time anyone reports it, the regression has been live for weeks.

Evaluation is the layer that catches those regressions before they ship. This article covers the metrics that actually tell you something, with runnable Python, and the minimum test-set recipe to get started. Prerequisite reading: the RAG pipeline walkthrough ships the four-step loop this article plugs into.

The two levels of RAG evaluation

A RAG system has two independent failure modes, so it needs two independent sets of metrics:

Retrieval quality — given a question, did the retriever return the chunks that actually contain the answer? If not, no amount of LLM cleverness can produce the right answer.
Generation quality — given the retrieved chunks (whatever they are), did the LLM produce a correct, faithful answer? A fluent answer that invents facts the chunks don't support is a generation failure, not a retrieval failure.

Evaluating only one layer misses half the problem. If retrieval is bad but generation masks it with plausible-sounding filler, the user sees "correct-looking wrong answers" and loses trust before anyone notices the metric that would have surfaced it.

Retrieval metrics

Precision@k

Of the top-K chunks the retriever returned for a question, how many are relevant?

def precision_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    top_k = retrieved_ids[:k]
    if not top_k:
        return 0.0
    hits = sum(1 for doc_id in top_k if doc_id in relevant_ids)
    return hits / len(top_k)

Example: top-5 retrieved = [A, B, C, D, E], relevant = {A, C}. Precision@5 = 2/5 = 0.4.

Use it when the context window is limited and the LLM gets confused by irrelevant chunks. Higher precision means less noise in the generation prompt.

Recall@k

Of all the relevant chunks that exist for a question, how many did the retriever surface in the top-K?

def recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    top_k = set(retrieved_ids[:k])
    if not relevant_ids:
        return 1.0  # nothing to find — vacuously perfect
    hits = len(top_k & relevant_ids)
    return hits / len(relevant_ids)

Example: top-5 retrieved = [A, B, C, D, E], relevant = {A, C, F}. Recall@5 = 2/3 ≈ 0.67 (missed F).

Use it when the answer requires combining multiple sources. If recall@k is low, the LLM is working with incomplete information and may fabricate the missing piece.

Mean Reciprocal Rank (MRR)

How high up the ranking does the first relevant chunk appear?

def reciprocal_rank(retrieved_ids: list[str], relevant_ids: set[str]) -> float:
    for i, doc_id in enumerate(retrieved_ids):
        if doc_id in relevant_ids:
            return 1.0 / (i + 1)
    return 0.0


def mean_reciprocal_rank(evaluations: list[tuple[list[str], set[str]]]) -> float:
    if not evaluations:
        return 0.0
    return sum(reciprocal_rank(r, rel) for r, rel in evaluations) / len(evaluations)

Example: first relevant chunk at rank 1 → RR = 1.0; at rank 3 → RR = 1/3 ≈ 0.33; not retrieved → RR = 0.

MRR is the right metric when the application only shows the top result to the user (search autocomplete, "best match" chat answers). It's less useful for multi-document synthesis where several top chunks matter.

Normalised Discounted Cumulative Gain (nDCG)

The retrieval metric most forgiving of what order relevant chunks appear in — but rewards higher positions. Useful when some relevant chunks are more relevant than others (graded relevance, not binary).

import math


def dcg(relevances: list[float]) -> float:
    """Discounted Cumulative Gain. relevances[i] = graded relevance of result i."""
    return sum(rel / math.log2(i + 2) for i, rel in enumerate(relevances))


def ndcg_at_k(retrieved_ids: list[str], relevance_grades: dict[str, float], k: int) -> float:
    """Normalised DCG@k. relevance_grades = {doc_id: grade}, grade >= 0."""
    actual = [relevance_grades.get(doc_id, 0.0) for doc_id in retrieved_ids[:k]]
    ideal = sorted(relevance_grades.values(), reverse=True)[:k]
    if not ideal or sum(ideal) == 0:
        return 0.0
    return dcg(actual) / dcg(ideal)

nDCG = 1.0 means the retrieved ranking matches the ideal ranking. nDCG = 0 means nothing relevant was retrieved.

Which metric to pick

A defensible default for most production RAG: track recall@k (did we find all the needed context?) and precision@k (how much noise in that context?). Add MRR if the UI only shows the top result. Add nDCG only if relevance is actually graded, not binary — which is rare in practice.

Generation metrics

Retrieval metrics judge the retriever. Generation metrics judge whether the LLM used the retrieved context correctly.

Faithfulness

Did the generated answer stick to the retrieved context, or did it invent claims the chunks don't support? Faithfulness is the metric that catches hallucination in a RAG system where retrieval was fine.

The honest way to measure faithfulness at scale is LLM-as-judge: a second model inspects the generated answer and the retrieved context, and flags claims in the answer that aren't supported by the context. Lightweight proxy for small test sets:

def claims_in_answer_covered_by_context(answer: str, context: str) -> float:
    """
    Trivial version — split the answer into sentences and check whether each
    sentence has a token-overlap with the context above a threshold.
    Not accurate enough for production; use an LLM judge for anything serious.
    """
    import re
    sentences = re.split(r"(?<=[.!?])\s+", answer.strip())
    if not sentences:
        return 1.0

    context_tokens = set(context.lower().split())
    covered = 0
    for s in sentences:
        s_tokens = set(s.lower().split())
        if not s_tokens:
            continue
        overlap = len(s_tokens & context_tokens) / len(s_tokens)
        if overlap > 0.4:
            covered += 1
    return covered / len(sentences)

Use this proxy to spot the worst drift. For reliable faithfulness in production, use LLM-as-judge with a prompt tuned to your domain.

Answer relevance

Did the answer address the question that was asked, or did it address an adjacent question? A hallucinated answer that happens to match some retrieved context can still be irrelevant.

Measured by prompting a judge model: "Given this question and this answer, rate how directly the answer responds, 1 to 5." Store the average per test query, track across releases.

Context recall + context precision

The two retrieval-adjacent metrics that look at how well the generation step used the context:

Context recall: of all the relevant information in the retrieved context, how much appeared in the answer? Low context recall = the LLM retrieved the answer but didn't surface it.
Context precision: of the retrieved chunks, how many were actually used in generating the answer? Low context precision = the retriever is noisy and the LLM is working harder than it should.

Both are classically measured with LLM-as-judge over a test set, and are the metrics that Ragas and similar libraries report by default.

Building a minimum test set

A test set is a list of questions with the "correct" answer for each, plus (for retrieval metrics) a set of chunk IDs that should have been retrieved. The smallest useful test set has roughly 20-50 questions covering the obvious categories of query the production system will see.

Minimum shape:

from dataclasses import dataclass


@dataclass
class EvalQuery:
    question: str
    expected_answer: str
    relevant_chunk_ids: set[str]  # Chunks that contain the answer
    category: str  # Tag for slicing results — "pricing", "troubleshooting", etc.


TEST_SET = [
    EvalQuery(
        question="What is the refund window after purchase?",
        expected_answer="7 days from the date of purchase.",
        relevant_chunk_ids={"chunk_42"},
        category="policy",
    ),
    EvalQuery(
        question="Which Python version is required?",
        expected_answer="Python 3.10 or later.",
        relevant_chunk_ids={"chunk_17", "chunk_18"},
        category="setup",
    ),
    # ... 18 more
]

Where the questions come from matters more than how many. Three defensible sources:

Real user queries from logs — if the system is already live, the top-50 most common actual questions are the right test set. Resolve "correct answer" by hand.
Support ticket questions — most teams have an archive of "customer asked X" questions with a resolved answer. Those are gold.
Hand-written by whoever knows the corpus best — fastest to start, but biased toward the writer's mental model of what users ask.

The handbook's Chapter 4 covers a five-step human review protocol for assembling the first 50 queries, and the pattern for growing the test set from production logs. For a free-article minimum, 20 hand-written queries covering 3-5 categories is enough to catch obvious regressions.

A simple pytest harness

Wire the metrics into pytest so the eval runs on every PR. Minimum version:

import pytest

from your_rag_app import rag_pipeline  # your retrieve + generate function
from your_eval_set import TEST_SET
from your_metrics import precision_at_k, recall_at_k


@pytest.fixture(scope="session")
def predictions():
    results = []
    for query in TEST_SET:
        retrieved_ids, answer = rag_pipeline(query.question)
        results.append({
            "query": query,
            "retrieved_ids": retrieved_ids,
            "answer": answer,
        })
    return results


def test_retrieval_recall_above_threshold(predictions):
    recalls = [
        recall_at_k(p["retrieved_ids"], p["query"].relevant_chunk_ids, k=5)
        for p in predictions
    ]
    avg = sum(recalls) / len(recalls)
    assert avg > 0.75, f"Average recall@5 = {avg:.3f}, below 0.75 threshold"


def test_retrieval_precision_above_threshold(predictions):
    precisions = [
        precision_at_k(p["retrieved_ids"], p["query"].relevant_chunk_ids, k=5)
        for p in predictions
    ]
    avg = sum(precisions) / len(precisions)
    assert avg > 0.30, f"Average precision@5 = {avg:.3f}, below 0.30 threshold"

Thresholds (0.75 recall, 0.30 precision) are starting points — tune to the system's current baseline, then enforce no regression below it. Run on every pull request so a chunking tweak or embedding model change can't silently break retrieval quality.

Tooling landscape in 2026

Three evaluation libraries dominate for RAG:

Ragas (pip install ragas) — the de facto standard for RAG-specific metrics. Ships faithfulness, answer relevance, context recall, context precision out of the box, using an LLM judge. Lowest-friction way to add generation metrics.
DeepEval (pip install deepeval) — pytest-native. Metrics look like unit tests, which is nice when CI integration is the priority. Covers RAG and non-RAG LLM apps.
Promptfoo (npm install -g promptfoo or Python API) — config-driven YAML eval sets, side-by-side model comparison, caching built in. Good when the main thing being evaluated is prompt + model choice, not retrieval.

Each has trade-offs around LLM-judge cost, metric calibration, and how it integrates with pytest. The handbook's Chapter 4 includes a decision table for picking between them based on team shape and existing CI.

What this article deliberately does NOT cover

Evaluation in production has a longer tail than the metrics above. The parts not covered here, and where to find them:

The four tested LLM-as-judge prompts that calibrate reliably against human review, with the measurement protocol for calibration.
The full pytest + GitHub Actions CI harness that runs on every PR without blowing up LLM API bills.
How to tell real metric movement from statistical noise on a 50-query test set — the Wilcoxon signed-rank procedure that avoids false positives.
The five-step human review protocol for building the first 50-query test set from scratch when there are no production logs yet.
Growing the test set from production logs — the sampling strategy that keeps the test set representative over time.
Ragas vs DeepEval vs Promptfoo — the decision table based on team shape, LLM-judge cost, and existing CI.

Those live in Chapter 4 of The RAG Production Handbook — launching November 9, 2026. First 20 buyers at ₹699 (regular launch ₹999, rising to ₹1,499 on Dec 1).