詞彙重疊檢查在偵測 RAG 幻覺方面勝過 DeBERTa NLI 模型
A word-overlap check beat a DeBERTa NLI model at spotting RAG hallucinations
在 RAGTruth 的 900 條測試回應中,詞彙重疊檢查偵測 RAG 幻覺的 AUROC 為 0.778,高於 DeBERTa NLI 模型的 0.716。詞彙檢查在全部 2,700 條測試回應中的 AUROC 為 0.785,而集成檢測器整體表現未勝過單獨使用詞彙檢查。作者建議把廉價檢測器用作分流訊號,而非封鎖答案的門檻。
I expected the neural entailment model to win. It didn't.
I wanted to know how far the cheap, no-API-call hallucination checks get on real RAG output before you need an LLM judge or a human. So I built three small detectors, ran them on RAGTruth, and compared them. The one that costs microseconds per response beat the one that costs about seven seconds on a CPU.
The setup
RAGTruth (Niu et al., ACL 2024, MIT licence) has about 18,000 responses from six LLMs to retrieval-grounded prompts, in three tasks: question answering, summarisation and data-to-text. Humans annotated hallucinated spans. I call a response hallucinated if it has at least one annotated span.
Each detector takes a source and a response and returns a score in [0, 1]:
- lexical: the share of content tokens in the response (lowercased, stopwords and punctuation removed, numbers kept) that never appear in the source.
-
numeric: the share of numbers in the response not found in the source, after normalising
1,000/1000/1k. -
nli: SummaC-style. Each response sentence is checked with
cross-encoder/nli-deberta-v3-smallagainst its three closest source sentences (by token overlap) and their concatenation. The response scores as its least-supported sentence. - ensemble: logistic regression on the three scores.
Thresholds and ensemble weights were fit on a stratified train sample of 299 responses and then frozen. Everything below is on a stratified test subsample of 900 of the 2,700 test responses (300 per task), with a 35.3% hallucination base rate.
The result
| detector | AUROC (95% CI) | precision | recall | F1 |
|---|---|---|---|---|
| lexical | 0.778 (0.746-0.806) | 0.476 | 0.858 | 0.612 |
| nli | 0.716 (0.681-0.747) | 0.476 | 0.836 | 0.607 |
| numeric | 0.535 (0.503-0.566) | 0.353 | 1.000 | 0.522 |
| ensemble | 0.766 (0.737-0.797) | 0.450 | 0.934 | 0.607 |
The lexical and numeric detectors are cheap enough to also run on all 2,700 test responses. Lexical gets AUROC 0.785 there, so the subsample isn't flattering it. The intervals for lexical and NLI barely overlap on the subsample, but I'd treat the gap as "lexical is at least as good here", not as a law of nature.
The ensemble, fit on 299 responses, didn't beat lexical alone overall. It only came out ahead on QA.
Why the cheap one wins (my read, not proven)
Most RAGTruth hallucinations add content: a name, a claim, a plausible-sounding detail the source never mentions. A check for "new vocabulary" catches a lot of that for free. The NLI model has a harder job, and my setup gives it a hard time in places: it only sees the top three source sentences, and its probabilities are saturated near 0 and 1 (the tuned threshold ended up at 0.9987).
On data-to-text, where the source is flattened JSON, NLI is at chance:
| task | hallucinated | lexical | nli | ensemble |
|---|---|---|---|---|
| QA | 20% | 0.708 | 0.706 | 0.740 |
| Summary | 20% | 0.616 | 0.627 | 0.582 |
| Data2txt | 66% | 0.661 | 0.496 | 0.663 |
AUROC within each task, 300 responses each. Note how much lower the per-task numbers are than the pooled 0.78: part of the pooled score comes from the detectors picking up on the task itself, since base rates differ so much between tasks.
The per-model picture is mixed too. NLI is the best detector on Mistral 7B instruct (0.812 vs 0.688 for lexical) and the worst on GPT-4 (0.689 vs 0.808). With 150 responses per model the intervals are wide, so read the pattern, not the third decimal.
The two kinds of detector also fail differently. In examples.json there are cases where NLI flags a clean-looking fabrication ("will make it the longest F1 circuit ever built", lexical score 0.18, NLI 0.9997) and cases where NLI lets plausible boilerplate through that the lexical check catches. That's the argument for combining them, and the ensemble result above is the argument against assuming it just works.
The caveats, because there are several
Precision is poor. At the threshold tuned for F1, the lexical detector catches 85.8% of hallucinated responses with precision 0.476. Working that through with the 35.3% base rate, about half of the correct responses get flagged too (this is my arithmetic from the table, not a number the benchmark reports directly). A flag is roughly a coin flip. Predicting "hallucinated" for everything gives F1 0.522, so an F1 of 0.61 is a modest gain, and AUROC is the fairer yardstick.
Thresholds are tuned on 299 responses to maximise F1. They won't transfer to your data. The numeric detector's best threshold was 0.0, which flags everything with a number missing from the source, so its row is just the all-positive baseline.
NLI ran on a subsample. It takes about 7.3 seconds per response on a loaded 4-thread CPU, so the full run took 2 h 33 min and I used 900 test responses instead of 2,700.
I did not run an LLM judge. The interface is in the repo, but there are no results for it, so none of this says anything about how these cheap checks compare with an LLM-as-judge. I'd expect the judge to be more accurate and also more expensive, but that's an expectation, not a finding.
Scope. English only, three tasks, 2023-era generator models, and response-level labels (span positions are ignored). I also didn't fine-tune anything on RAGTruth, which would flatter the numbers on this benchmark.
What I'd do with it
Treat a cheap detector as a triage signal, not a gate. If you block or rewrite every flagged answer, you'll degrade a lot of good ones. If you use the score to decide which answers a person reads first, it's useful. And the ranking between detectors changed with task and model, so the only way to know what works on your traffic is to label a few hundred of your own responses and measure.
Reproduce it
Code: github.com/protforge-ai/groundcheck
git clone https://github.com/protforge-ai/groundcheck && cd groundcheck
pip install -e ".[test]"
python -m pytest
groundcheck bench --seed 0
bench downloads the RAGTruth files, scores the train and test samples, and writes metrics.json, by_task.csv, by_model.csv, test_scores.csv and examples.json into results/. NLI scores are cached, so an interrupted run resumes. To score one response against its source:
groundcheck score --source source.txt --response response.txt --detector nli
--detector is one of lexical, numeric, nli. It prints JSON with a response score and a per-sentence verdict. The default --threshold of 0.5 is not tuned, so tune it on your own data.
The code is MIT licensed, built by ProtForge. The full write-up, with a per-model table, is at protforge.com/research/hallucination-detectors.
If you've run something similar on a different corpus and got a different ranking, I'd like to hear it.
Citation
Niu et al. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. ACL 2024. arXiv:2401.00396.
Drafted by an AI assistant from the benchmark results and reviewed by me. Every number comes from results/metrics.json in the repo.
來源:DEV Community · dev.to