臨牀智能體系統的 Evidence-Aware RAG 評估：為何標註覆蓋率決定檢索評分

Your RAG system's retrieval scores depend on how much of your corpus you've annotated, not on retriever quality alone. A new paper from a clinical AI team demonstrates this measurement artifact in a breast lesion management system and proposes an evidence-aware evaluation protocol that separates annotation coverage from system quality.

The problem surfaces when you build evals for high-stakes agent systems. You can't exhaustively label every relevant passage in a large corpus. Your ground truth becomes a sample, and your retrieval metrics reflect that sample's completeness as much as your system's performance.

The Annotation Coverage Problem

Standard RAG evaluation assumes your ground truth annotations are complete. You label relevant chunks for each query, run your retriever, and measure precision and recall. But in practice:

You annotate the most obvious evidence first
Edge cases and secondary sources get missed
Your retrieval scores plateau because you've run out of labeled targets, not because your system has reached its ceiling

The clinical team tested this with an 18-document corpus and 18 queries about breast lesion management. Initial curation produced 25 evidence spans mapped to 36 chunk targets. After a second pass, they found 37 spans and 51 chunks without changing retrieval rankings. Recall@3 stayed at 0.94, but the denominator grew by 41.7%. The system retrieved the same chunks, but the eval dataset revealed more relevant targets.

In any RAG system over a corpus larger than a few dozen documents, incomplete annotation is the norm. The paper quantifies this: a 48% increase in evidence spans (25 to 37) and a 42% increase in chunk targets (36 to 51) occurred between curation rounds, with zero change to retrieval output.

Evidence-Aware Evaluation Protocol

The protocol treats annotation coverage as a first-class variable. Instead of pretending your ground truth is exhaustive, you track:

Evidence spans: The actual text passages that answer each query
Chunk targets: All chunks containing any part of those spans
Annotation rounds: How many curation passes you've completed
Coverage estimates: What fraction of true positives you've likely labeled

For overlapping chunks, the protocol considers all chunks with equivalent evidence as relevant. If your chunking strategy creates three 384-token chunks that each contain the same critical sentence, all three count as correct retrievals.

Retrieval Pipeline Architecture

The system uses a three-stage retriever:

# Pseudocode illustrating three-stage retrieval pipeline
# Stage 1: Parallel sparse and dense retrieval
bm25_results = bm25_index.search(query, top_k=100)
dense_results = embedding_model.search(query, top_k=100)

# Stage 2: Merge and deduplicate
# merge_results() is a custom function that unions result sets
# and removes duplicate chunk IDs
candidate_pool = merge_results(bm25_results, dense_results)

# Stage 3: Cross-encoder reranking
reranked = cross_encoder.rerank(
    query=query,
    candidates=candidate_pool,
    top_k=5
)

The cross-encoder reranker consistently outperformed single-stage retrievers. With 384-token chunks and 20% overlap, it retrieved accepted evidence in the top three results for 17 of 18 queries.

Chunking Strategy Trade-offs

The team tested chunk sizes from 128 to 1024 tokens with overlaps from 0% to 20%. Results from the paper's evaluation:

Chunk Size	Overlap	Recall@3	R-Precision	Notes
128 tokens	0%	0.78	0.72	Fragments evidence spans
384 tokens	20%	0.94	0.94	Selected configuration
512 tokens	0%	0.89	0.83	Misses boundary cases
1024 tokens	20%	0.83	0.78	Dilutes relevance signal

The 384-token chunk with 20% overlap balances two failure modes. Smaller chunks fragment evidence and hurt recall. Larger chunks dilute the relevance signal and hurt precision. Overlap catches evidence that spans chunk boundaries.

Generation Layer Evaluation

Retrieval is only half the system. The generation layer was evaluated separately using:

DeepSeek-v4-pro as an LLM judge for claim-level metrics
BLEU and ROUGE for surface-level text similarity
K values from 1 to 5 to test context window utilization

The separation matters for observability. When your agent produces a bad answer, you need to know whether retrieval failed (wrong chunks in context) or generation failed (right chunks, wrong synthesis).

State Management and Curation Flow

The evidence-aware protocol requires tracking annotation state across curation rounds. The team's workflow:

Initial annotation: Domain expert labels obvious evidence spans
Retrieval run: System retrieves top-K chunks for each query
Gap analysis: Compare retrieved chunks to labeled spans
Iterative curation: Label newly discovered evidence in retrieved results
Coverage estimation: Track how many rounds pass without finding new evidence

This creates a feedback loop between system performance and ground truth quality. Your retrieval system becomes a discovery tool for improving your eval dataset.

Failure Modes and Observability

The protocol exposes three distinct failure modes:

Retrieval failure: System doesn't surface relevant chunks. Fix with better embeddings, reranking, or chunking strategy.

Annotation gap: System retrieves relevant chunks that aren't labeled. Fix with more curation rounds.

Evidence insufficiency: Corpus doesn't contain the answer. Fix with corpus expansion or query refinement.

Standard RAG evals conflate these. A low recall score could mean any of the three. Evidence-aware evaluation separates them.

Deployment Considerations

For production systems:

Annotation budget: Plan for multiple curation rounds. First-pass annotations are incomplete by default.
Metric interpretation: Track annotation coverage alongside retrieval scores. A 0.85 recall with 50% coverage is different from 0.85 with 95% coverage.
Continuous curation: Use production retrieval logs to discover unlabeled evidence. Your system will find edge cases you missed.
Domain expertise: Clinical systems need radiologists for iterative curation. Legal systems need attorneys. Financial systems need compliance officers. Annotation quality matters more than quantity.

Security and Compliance Boundaries

Clinical RAG systems operate under HIPAA constraints. The evaluation protocol doesn't change compliance requirements, but it does affect audit trails:

Evidence spans must be traceable to source documents
Annotation decisions need provenance (who labeled what, when)
Retrieval logs may contain PHI and require encryption at rest
LLM judge outputs (DeepSeek in this case) must stay within your security boundary

If you're using an external LLM for evaluation, you're sending query-chunk pairs outside your perimeter. Consider local models or anonymization pipelines.

Technical Verdict

Use evidence-aware evaluation when:

Your corpus is too large to exhaustively annotate
You're building high-stakes systems (medical, legal, financial) where eval quality matters
You need to separate retrieval performance from annotation completeness
You have domain experts (radiologists, attorneys, compliance officers) available for iterative curation

Avoid it when:

Your corpus is small enough to fully annotate in one pass (under 50 documents)
You're prototyping and need fast iteration over rigorous measurement
You don't have domain expertise to validate evidence spans
Your use case tolerates approximate evals (internal tools, low-risk applications)

The protocol adds overhead. You're tracking annotation rounds, coverage estimates, and evidence provenance. But for clinical agents and other high-stakes systems, the alternative is shipping evals that lie about system quality because they conflate retrieval performance with annotation gaps.