如何評估 RAG 檢索質素：離線 Recall 與點擊訊號比較，為何相關性標註更合適

Retrieval quality is constrained by what the evaluator calls relevant. For a system that watches documentation pages, retrieves supporting passages after a detected change, and alerts with a grounded diff, I choose judged relevance labels over click proxies for the release gate. Short answer: compute Recall@K against a versioned set of question-to-relevant-passage labels, fail the build when an agreed slice regresses, and keep clicks as a separate online signal. This choice holds when missing a supporting passage is costlier than returning an extra candidate; it does not settle ranking quality, citation correctness, or answer faithfulness.

A plausible incident is narrow: a page changes, the monitor detects it, and the retriever still returns a nearby but stale passage. The alert can look fluent while citing text that does not support the diff. The invariant is harsher than "the result looked reasonable": every evaluation query needs an explicit set of acceptable passage IDs tied to the corpus version being tested. No label, no defensible denominator.

How should you evaluate RAG retrieval quality with offline recall?

Recall@K asks what fraction of the relevant items for a query appear in the first K retrieved items. If the relevant set is R and the top-K result set is S, the per-query score is |R intersection S| / |R|. The original RAG paper evaluates retrieval as part of a knowledge-intensive generation system, but this metric should remain a retrieval test: generation can conceal a miss by producing plausible language.

The first trap is averaging everything together. Suppose an illustrative fixture has 20 questions: 15 about headings that barely changed and five about removed configuration constraints. A strong score on the easy majority can cover a complete miss on removals. Report macro recall, but also report slices for change type, source template, language, and freshness window. The numbers are illustrative; the slicing rule is the point.

The second trap is treating one relevant passage as the only relevant passage. A changed API limit might be supported by a table row and by explanatory prose. If either passage grounds the alert adequately, the judgments should say so. Conversely, marking an entire page relevant makes a weak chunker look better than it is. Labels belong at the same unit the retriever returns.

One miss matters.

Freeze the corpus.

I would make the release rule asymmetric: block on a regression in the high-risk removal slice even when the global mean is flat, because a false sense of continuity is worse than an alert that includes one extra supporting chunk. That is an SLO choice, not a mathematical property of Recall@K. Write it down before comparing runs.

The offline harness

This Go program reads a JSON fixture, deduplicates retrieved IDs, evaluates several K values, and prints macro Recall@K. It rejects malformed cases instead of quietly changing the denominator.

package main

import (
    "encoding/json"
    "errors"
    "fmt"
    "log"
    "os"
)

type Case struct {
    QueryID string `json:"query_id"`
    Relevant []string `json:"relevant"`
    Retrieved []string `json:"retrieved"`
}

func recallAtK(c Case, k int) (float64, error) {
    if c.QueryID == "" || len(c.Relevant) == 0 {
        return 0, errors.New("query_id and relevant passages are required")
    }
    relevant := make(map[string]struct{}, len(c.Relevant))
    for _, id := range c.Relevant {
        if id == "" { return 0, errors.New("passage IDs cannot be empty") }
        relevant[id] = struct{}{}
    }
    limit := k
    if limit > len(c.Retrieved) { limit = len(c.Retrieved) }
    seen := make(map[string]struct{}, limit)
    hits := 0
    for _, id := range c.Retrieved[:limit] {
        if _, duplicate := seen[id]; duplicate { continue }
        seen[id] = struct{}{}
        if _, ok := relevant[id]; ok { hits++ }
    }
    return float64(hits) / float64(len(relevant)), nil
}

func main() {
    if len(os.Args) != 2 { log.Fatal("usage: recall cases.json") }
    data, err := os.ReadFile(os.Args[1])
    if err != nil { log.Fatal(err) }
    var cases []Case
    if err := json.Unmarshal(data, &cases); err != nil { log.Fatal(err) }
    if len(cases) == 0 { log.Fatal("at least one case is required") }
    for _, k := range []int{1, 3, 5, 10} {
        total := 0.0
        for _, c := range cases {
            score, err := recallAtK(c, k)
            if err != nil { log.Fatalf("%s: %v", c.QueryID, err) }
            total += score
        }
        fmt.Printf("Recall@%d %.3f\n", k, total/float64(len(cases)))
    }
}

Use passage IDs that are stable within a corpus snapshot, then store the snapshot identifier beside the evaluation run. Otherwise a rechunking deploy can turn yesterday's labels into unmatched IDs and appear to be a retrieval collapse. The harness should also verify that every judged passage exists in the tested snapshot. That check is omitted here because storage interfaces differ, but it belongs before score calculation in production.

The failure chain deserves a dry run before anyone trusts the gate: take one labeled passage, remove it from a copied snapshot, rebuild the index, and confirm that the relevant slice falls while unrelated slices remain stable. Then restore the passage under a new chunk ID and confirm that the existence check rejects the old judgment instead of reporting a retrieval miss. Finally, update the judgment deliberately and record that review as a new evaluation-set revision. This is a controlled fixture, not a benchmark claim, and it tests the plumbing that a clean Recall@K formula cannot test by itself.

Gate the slice.

For capacity planning, multiply queries x candidate K values x retrieval variants x corpus snapshots. A set of 2,000 judged queries, four K values, three configurations, and two snapshots implies 48,000 scored query/configuration combinations before reranking experiments are added. Cache immutable retrieval results by snapshot and configuration; do not hide live index calls inside a supposedly offline test.

Labels or behavioral proxies

Clicks are tempting because they arrive without a judging project. For a change-alert system, however, a click can mean the title was alarming, the snippet was confusing, or the user needed to inspect a bad alert. Absence of a click can mean the alert contained enough information. That ambiguity makes clicks useful for discovering evaluation cases, not for defining offline relevance on their own.

Signal	What it supports	Operational liability	My use
Human passage judgments	A reviewable relevance denominator	Labeling cost and disagreement	Release gate
Click or open events	Broad behavior trends	Position, presentation, and intent bias	Case discovery
Synthetic judgments	Fast coverage expansion	Evaluator drift and correlated errors	Triage before human review

This is also a buy-versus-build boundary. Buying annotation workflow can reduce coordination work; building a small internal format keeps the contract portable. Neither option removes the need for written relevance guidance, adjudication of disagreements, corpus-version pinning, and audit samples. I would buy workflow before buying the definition of relevance.

Do not tune K until the judgments survive review. Increasing K often raises recall mechanically, but it also sends more material to reranking and generation, expands latency and token demand, and can dilute citation selection. Plot recall against K, then choose the smallest K that meets the slice SLO while remaining inside the downstream capacity budget.

Prevent the stale-citation path

The preventative path begins when a page snapshot changes. Normalize and chunk the new snapshot, retain its immutable version, map or rejudge affected labels, run retrieval against that exact version, and compare the candidate run with the accepted baseline. Deploy only after the relevant slices pass. During serving, carry snapshot and passage IDs through retrieval, answer construction, and the citation shown in the alert.

That lineage matters because Recall@K can pass while the final citation points elsewhere. Track separate checks for retrieval recall, citation entailment, diff accuracy, and end-to-end freshness. A single composite score makes the on-call diagnosis slower: a red number does not tell the responder whether indexing, retrieval, reranking, or citation assembly moved.

For observability, emit the evaluation-set revision, corpus snapshot, retriever configuration hash, K, slice, and score as run metadata. Alert on policy violations rather than tiny statistical movement. A test set with ten removal questions changes its slice score by ten percentage points per miss; pretending that such a jump has fine precision produces noisy gates. Expand the judged set or use confidence intervals before tightening the threshold.

Where Recall@K stops helping

Recall@K is the wrong primary gate when there is no enumerable relevant set, when the task rewards exactly one best result, or when the dominant risk is unsupported generation after retrieval succeeds. Mean reciprocal rank or nDCG can capture ordering preferences; citation and answer evaluation require their own evidence-based tests. Do not infer those qualities from recall.

It is also incomplete for newly changed pages. An offline set drawn only from last month's corpus cannot measure behavior on today's novel layouts or vocabulary. Add reviewed cases from production changes on a fixed cadence, but keep a frozen core set so trend lines retain meaning. Fresh coverage and comparability pull in opposite directions, so use two suites rather than quietly rewriting one.

For page-change alerts, judged labels win as the retrieval release gate because they make misses inspectable and tie every denominator to a corpus version. Clicks and synthetic judgments can help find blind spots. They should not get the final word. The practical boundary is clear: use Recall@K to prove candidate coverage, slice it by operational risk, and require separate evidence for ordering, citations, freshness, and generated claims.