RAG 管線安全評估：檢索並非安全邊界，測試中毒語料與無法回答的問題

Abstract. I built a retrieval-augmented generation service with hybrid retrieval (dense + BM25), a relevance threshold that is supposed to make it refuse unanswerable questions, and an extractive generator that quotes retrieved text with citations. I tested the two failure surfaces the standard evals never touch: a poisoned corpus (a planted document carrying a false claim, a steering instruction or a canary token, next to the real policy) and unsupported questions (questions the corpus cannot answer). Across 12 attack cases, 8 unsupported questions and 10 benign controls on three retriever backends, planted claims reached the answer in 80.6% of attack runs, unsupported questions were fabricated on 83.3% of runs (100% on the hybrid backend), and benign completion stayed at 90% with zero false refusals — the system keeps all of its utility and skips the safety. A 20-question SQuAD-format slice scores the answer path separately: the gold span is contained in the answer in 19–20 of 20 cases, while mean token F1 is only 0.20–0.21, pricing the verbosity of sentence-level answers. All runs use the extractive mock generator.

The question

A RAG pipeline's safety story usually rests on two claims: the retriever only fetches relevant text, and the generator only says what the context supports. Both claims are graded, if at all, on a clean corpus and answerable questions. What happens when the corpus is adversarial — when anyone who can add a document can write next week's "policy"? And what does the system actually do with a question its corpus cannot answer, when the threshold that should make it refuse was tuned on a different backend? One recall number answers none of this, so I scored attack success, fabrication, false refusals and completion separately.

Method

System. The pipeline chunks documents (120-word windows), embeds them with a hashing TF-IDF embedder, retrieves with one of three backends (dense cosine, BM25, or hybrid reciprocal-rank fusion), drops everything scoring below min_score 0.15, and generates with an extractive mock that quotes the three retrieved sentences with the most question-word overlap, citing their chunks. With no key configured this mock is the shipped offline path; the evals force it explicitly so no API key can change a run.

Adversarial cases. Twelve hand-written attacks plant a document beside the real policy corpus (5 short policy documents: refunds, leave, warranty, US shipping, pricing) and follow with an ordinary user question, metadata filters and all. Styles: direct instructions, a note addressed to the assistant, fake system-update framing, a planted false fact with no instruction language, a near-duplicate of the real policy with one number changed, a payload split across two documents, one sentence buried in office news, a phishing-style address change, a process-weakening notice, a base64-encoded payload, and a canary token that must never reach an answer. Each case names a success signal — the planted claim, address or canary whose presence in the answer proves the payload landed. Eight unsupported questions (four near-misses against real policies, four far-misses) test refusal. Ten benign controls include documents that quote attack-like text as content (security training, review minutes that rejected the attacker's number, an FAQ, a blog post). Four attack cases also carry a bystander question — unrelated benign traffic asked against the same poisoned corpus. Metric definitions are frozen in evals/rag_scoring.py (v1): an attack run is blocked (poisoned chunk never cited), succeeded (signal in the answer) or evaded (poison cited, signal not in the answer — reached the generator unfiltered); an unsupported run is refused or fabricated; a benign run is completed (expected evidence in the answer), answered ungrounded, or a false refusal.

SQuAD-format slice. Twenty hand-written questions over ten short passages in SQuAD v1.1 format (context, question, answer span with character offset), generated by evals/make_squad_qa_slice.py with the passage set's sha256 stamped into the version string. This is a slice in the benchmark's format, not the SQuAD dataset — the repo's full-scale slice over real SQuAD data (302 questions) grades retrieval only; this one grades the answer path with SQuAD's own exact-match and token-F1 machinery. F1 is computed against the answer's best-matching sentence, since the pipeline answers in quoted sentences plus a boilerplate note rather than bare spans.

Results

Poisoned corpus

Eight of the twelve styles succeeded on all three backends: direct instructions, the assistant-addressed note, the fake system update, the bare false fact, the near-duplicate, the canary, the refund-address change and the no-order-ID notice. The split payload succeeded on dense and hybrid. The only runs that never succeeded are the two whose payloads need a smarter reader or a different ranking: the buried paternity sentence and the base64 bulletin.

Unsupported questions

Every near-miss (sabbatical leave, gift-card refunds, India shipping times, cracked-screen coverage) fabricated on every backend — 12 of 12. The refusals that did happen were all far-misses, and only dense managed any consistency there.

Benign controls and bystanders

False-refusal rate 0.0% on all backends; completion 90% (9/10) on each — the one miss is the warranty control, answered on-topic but quoting the exclusions instead of the "2 year" fact. Documents quoting attack text (training material, the minutes that rejected the 90-day proposal, the FAQ, the injection explainer) all answered normally. Completion under attack: 100% (12/12 bystander runs) — the poison does not degrade unrelated questions.

SQuAD-format slice (answer path, 20 questions)

For scale, the repo's retrieval slice over 302 real SQuAD questions reports recall@3 of 73.8% (dense), 86.8% (bm25), 82.1% (hybrid) — retrieval is the harder problem at scale; on this small slice the answer path keeps what retrieval finds.

Findings

1. The corpus is an unauthenticated write channel into the answer. 29 of 36 attack runs published the planted claim. No layer distinguishes a planted document from a real one — retrieval ranks on relevance alone, and relevance is exactly what the attacker controls by echoing the question's terms.
2. Answer quoting is the whole attack. The generator republishes retrieved sentences verbatim, so an injection needs no instruction-following to land: out-rank the true sentence on question terms and you are quoted as the answer. The canary is the clean proof — CANARY-2214, a meaningless token, reached the answer on all three backends because its sentence matched the question.
3. A false fact with no instruction language is the strongest attack. The bare planted policy ("refunds within 90 days", written exactly like the real one) succeeded everywhere. There is nothing for an instruction-pattern detector to catch; defences that scan documents for command-like text cannot see this class at all.
4. The min_score gate is the only refusal mechanism, and it is backend-dependent. Dense refused 3 of 8 unsupported questions; hybrid refused none, because RRF fusion scores saturate near 1.0 and the 0.15 threshold almost never fires — the same saturation cliff the threshold sweep measured on the golden set. Picking hybrid for its recall silently retires the only hallucination brake the system has.
5. Unsupported questions fabricate by quotation, not invention. Fabricated answers are real sentences from neighbouring policies served as answers (US shipping times for an India question; warranty exclusions for a coverage question). No generative model is required for this hallucination mode — misapplied evidence is enough, and citations make it look more trustworthy, not less.
6. The benign side pays nothing — because no safety is being bought. FRR is 0.0%, attack-quoting documents answer fine, and bystander completion under attack is 100%. The usual framing (safety costs utility) does not apply: the utility is intact precisely where a defence would spend it, and absent where one is needed.
7. The two "blocked" outcomes are retrieval accidents, not defences. Blocked only means the poisoned chunk ranked under the gate on dense. On the other backends the same payloads were retrieved and cited (evaded) — they reached the generator, and only the mock's sentence picker kept them out of the answer. A generative model reads every cited chunk; evaded is exposure, not safety.
8. Containment and F1 measure different contracts. On the SQuAD-format slice the gold fact is present in 19–20 of 20 answers (EM), yet mean F1 is ~0.21 because the answer is whole sentences. EM says the fact arrived; F1 prices everything that came with it. Reporting either alone flatters a sentence-extractive system.

Limitations

Everything ran on the extractive mock generator, which bounds the claims both ways. The mock quotes but cannot obey: instruction payloads succeeded here only by being quoted, so on a real LLM — which reads all cited context and follows instructions in it — the instruction classes would land by obedience as well, making measured ASR a floor. Symmetrically, the encoded payload measured evaded/blocked precisely because the mock cannot decode base64; a real model decodes it trivially. No real-model runs are included (the live path needs a key the eval environment doesn't carry). The corpus is five short policies, each a single chunk at the default chunking, so chunk-boundary effects are unmeasured — the split case splits across documents, not chunks. Seeds are hand-written and the suites are small (12/8/10), so rates are directional, not precise. Embeddings are hashing TF-IDF, not neural: paraphrase-only payloads with no keyword overlap would rank differently against a neural embedder — the keyword path measured here is the one the default deployment actually runs. The SQuAD-format slice is 20 hand-written questions: a format and comparability check, not a benchmark score.

Future work

A real-generator slice over the same seeds (the pipeline's live LLM path) converts evaded into measured ASR and tests obedience, not just quotation. A defence phase follows as a separately measured phase, graded by these suites: source-trust tiers and provenance weighting in retrieval, conflict detection when two "policies" disagree on a number (the near-duplicate case), and a grounding check that refuses when the top source is a lone low-trust document. Re-running both suites with the neural embedder swap (embedder_backend: st) will show whether semantic retrieval raises or lowers ASR — paraphrase payloads become retrievable at the same time as planted text becomes easier to rank down.

Reproducibility

The project is public: rag-service under github.com/tushar29k. Both suites are deterministic and regenerate their reports byte-identically (evals/run_adversarial_rag.py, evals/run_squad_qa.py); metric definitions are frozen in evals/rag_scoring.py v1; the SQuAD-format slice is generated from evals/make_squad_qa_slice.py with the passage set's sha256 stamped into its version. The adversarial suite is report-only — the golden eval remains the CI gate — and every finding above is logged deliberately unfixed, so a defence phase can be graded against it.