本地 RAG 如何在證據不足時拒答
Local RAG That Knows When It Has No Evidence
文章提出一套本地 RAG 流程,要求答案以穩定來源 ID 引用直接支持的段落,並在資訊不足時返回 INSUFFICIENT_EVIDENCE。流程建議保留來源路徑、標題和內容雜湊,並用可回答、缺少事實及相關但不足的問題測試答案與拒答。相似度門檻應按自有數據校準,並在嵌入模型、語料或距離計算改變時重新測試。
A retrieval system can return a confident answer to a question its documents never address. It may have retrieved a vaguely related paragraph, or the model may fill a gap from general knowledge. Showing citations helps, but a citation is not proof: the source might not support the sentence beside it.
The fix is a workflow that makes evidence explicit. Give every indexed passage a stable source ID, require the answer to point to supporting passages, and permit a clear “not enough evidence” result. Then test that behavior with questions the system should not be able to answer.
Keep provenance with every passage
At minimum, store each passage with a stable identifier, source path, title, and the source’s content hash. A record might look like this:
{
"id": "handbook-2026-04:access#p3",
"path": "handbook/access.md",
"title": "Access reviews",
"content_sha256": "<hash of the indexed source>",
"text": "Temporary access expires after seven days unless renewed."
}
Replace the hash placeholder with the source digest. A citation should identify the version and passage that was indexed, so an edit to the handbook does not erase the evidence behind an earlier answer. Keep that source text available for citation rendering.
Ollama’s embedding endpoint accepts one string or a batch and returns vectors for semantic retrieval. Similarity helps rank candidate passages, but a high-ranking passage may still lack the requested fact. See the Ollama embeddings guide.
Write down an abstention contract
Before calling the answer model, create a prompt that makes the evidence boundary concrete:
Answer using only the passages below.
For each factual claim, include one or more passage IDs that directly support it.
If the passages do not contain enough information, set answer to
INSUFFICIENT_EVIDENCE and evidence to [].
Do not use outside knowledge to fill gaps.
Question: When does temporary access expire?
[handbook-2026-04:access#p3]
Temporary access expires after seven days unless renewed.
Answer format:
answer: <one sentence or INSUFFICIENT_EVIDENCE>
evidence: [<passage ID>, ...]
For the included passage, a supported answer is “After seven days unless renewed” with the listed ID. If asked “Who approved the policy?” the correct output is answer: INSUFFICIENT_EVIDENCE and evidence: []: the passage does not say. Flag malformed output, unknown IDs, and citations that fail to support their claims for review.
Prompt wording alone cannot force every model to abstain consistently. Add a retrieval gate calibrated on your own data. For example, inspect whether the top candidates are relevant and whether at least one passage contains the kind of evidence needed. Do not copy a similarity threshold from another project: embedding models, corpus, and distance calculation affect score distributions. Store the chosen policy and test it when any of those change.
Research such as Self-RAG explores retrieval relevance and reflection as ways to improve factuality. The workflow here uses a simpler application-side contract: retain source passages, make evidence IDs explicit, and test the abstention path.
Hold out questions before tuning
Create an evaluation set with three categories: answerable questions, unanswerable questions about missing facts, and misleading questions with a related but insufficient passage. Keep some questions out of prompt and threshold tuning. For each, record expected answerability, acceptable source IDs, and the unsupported fact to watch for.
An illustrative test set might include:
| Question | Expected | What it catches |
|---|---|---|
| When does temporary access expire? | Answer with handbook-2026-04:access#p3
|
Supported fact |
| Who approved the policy? | Abstain | Missing fact |
| Does renewal extend access forever? | Abstain | Over-reading a related sentence |
Run these cases through the application's retrieval and answer path. Check both whether it abstains on missing facts and whether it answers the question the passage does support. Record unnecessary abstentions alongside unsupported answers.
Save retrieved IDs, scores, prompt version, model identifier, and final answer for each run. If “Who approved the policy?” starts producing a name after an update, those records help locate the change in indexing, retrieval, or generation.
來源:DEV Community · dev.to