RAG 上下文預算指南：如何整理證據而不淹沒答案

When a RAG answer is wrong, the reflex is often to raise top_k. More retrieved chunks feels safer. In production, it can do the opposite: the one useful passage gets surrounded by repeated, stale, or merely related text, while the model has less room to answer.

A large context window is capacity, not a retrieval strategy. Treat the input as a RAG context budget: reserve room for the answer, retrieve widely, then admit only the evidence that earns its token cost. This guide shows a vendor-neutral policy you can implement around any vector store and model API.

The mistake: confusing retrieval recall with prompt admission

Retrieval and generation have different jobs. A retriever should find a broad candidate set so a good passage is not missed. The prompt assembler should build a small, diverse, attributable evidence set that a model can use.

Those are not the same top_k.

For example, a support assistant may retrieve 30 candidates to protect recall, rerank the best 12, and send four short, non-overlapping sections to the model. A policy question may need the newest authoritative policy plus one exception. A “list every affected region” question may need more coverage and a different answer shape.

Sending every candidate creates four problems:

• Attention dilution: Related passages repeat terms without answering the question.
• Unstable cost and latency: A broad request can quietly turn into a huge input.
• Crowded-out answers: The model has little output room for a careful explanation or citations.
• Weak debugging: A trace says “top 20 retrieved,” but not which evidence actually mattered.

Recent retrieval systems increasingly select useful highlights rather than whole pages. That direction is sensible, but it does not remove the engineering decision: your application still needs an explicit policy for what may enter the generation prompt.

Define the budget before tuning chunks

Start from the smallest context you want to support reliably, not from a model’s maximum window. Reserve fixed and variable space first:

The basic equation is:

evidence_budget = input_limit
                - instruction_tokens
                - conversation_tokens
                - answer_reserve
                - safety_margin

Use the provider tokenizer for the final count. A rough estimator is fine while choosing candidates, but do not assume words equal tokens or that chat-message formatting is free.

The budget is a product decision too. If the user asks for a comparison with sources, reserve enough output for distinctions and citations. If the task returns a short JSON classification, the answer reserve can be smaller. A single global budget hides that trade-off.

Admit evidence in stages, not by a fixed top-k

A practical context packer has four gates:

1. Retrieve candidates broadly. Apply tenant, access-control, language, and freshness filters before similarity search where possible. Never let a relevance score override an authorization decision.
2. Rerank against the actual question. Embedding similarity is a candidate generator, not proof that a passage answers the request. A cross-encoder, rules, or a small model can score query-passage relevance.
3. Remove redundancy. Adjacent chunks, copied help-center pages, and syndicated content often say the same thing. Keep the best representative and retain its source identity.
4. Pack to the remaining token budget. Prefer authority, answer coverage, recency when relevant, and source diversity—not score alone.

This means a lower-scoring chunk can win a place because it covers a missing part of a multi-part question. Likewise, a high-scoring duplicate should lose once its fact is already represented.

Make the query class explicit

One cap cannot serve every intent. Begin with a small classifier that chooses a policy, and log its decision:

Start with deterministic routing based on obvious cues and test data. You can later use a classifier, but its output should remain observable and easy to override. A wrong class should fail safely by returning a narrower answer, asking a question, or disclosing limits—not by stuffing every result into the prompt.

A small packing policy in Python

The example below accepts already-authorized, reranked candidates. It leaves a fixed answer reserve, rejects near-duplicates, and gives each source a small diversity allowance. It is intentionally simple: production systems also need timeout handling, tracing, and model-specific token counting.

from dataclasses import dataclass

@dataclass
class Chunk:
    id: str
    source_id: str
    tokens: int
    relevance: float
    novelty: float       # 0..1; lower means it repeats accepted evidence
    authority: float     # 0..1; derived from your source policy
    text: str

def pack_context(candidates, evidence_budget, per_source_limit=2):
    accepted, used, source_counts = [], 0, {}

    # Relevance gets us close; novelty and authority stop a score-only pack.
    ranked = sorted(
        candidates,
        key=lambda c: 0.60 * c.relevance + 0.25 * c.novelty + 0.15 * c.authority,
        reverse=True,
    )

    for chunk in ranked:
        if chunk.novelty < 0.20:
            continue
        if source_counts.get(chunk.source_id, 0) >= per_source_limit:
            continue
        if used + chunk.tokens > evidence_budget:
            continue
        accepted.append(chunk)
        used += chunk.tokens
        source_counts[chunk.source_id] = source_counts.get(chunk.source_id, 0) + 1

    return accepted, used

Do not mistake this heuristic for ground truth. The purpose is to make an admission policy visible and testable. Your novelty signal could be cosine-distance from accepted chunks, heading overlap, a deduplication hash, or an evaluator. Your authority signal should come from a source registry rather than an LLM’s opinion.

If one excellent section is larger than the budget, do not blindly slice it at a character boundary. Prefer a precomputed passage, a parent section with an anchored excerpt, or a targeted extraction step. Keep the citation identifier with every excerpt so the answer can point back to what it used.

Reserve room for the answer before assembling evidence

The most subtle failure is calculating evidence first and letting the answer take whatever remains. That makes answer quality depend on retrieval volume.

Instead, choose an answer contract first. A short factual answer may reserve 300 tokens. A troubleshooting response with diagnosis, safe steps, and cited evidence may reserve 900 or more. A structured output contract should reserve room for its full schema and error case.

Set an evidence ceiling below the model limit even if you could fit more. This makes cost and latency predictable, and it makes behavior portable when you later route a request to a smaller model. If the candidate pack does not fit, apply a known fallback:

• return the highest-authority answer with a scope note;
• ask the user to narrow an aggregation request;
• run a second retrieval pass for a missing sub-question; or
• summarize a trustworthy parent document, while preserving provenance.

Avoid silently dropping the last chunks. Tail-drop creates position bias and makes results depend on incidental sort order.

Keep instructions, evidence, and tool output separate

Context packing is also a safety boundary. A retrieved document is evidence, not an instruction. Label it as untrusted source material, keep it outside the system instruction channel, and do not let its text decide which tools are available. This matters even for an internal knowledge base: imported tickets, copied documentation, and web pages can contain text that looks like a command.

Use a small envelope for every admitted passage, such as a source ID, title, version or fetch time, access label, and the excerpt itself. The model can cite the ID, while application code resolves it to a safe link. Do not invite the model to invent an origin from raw text.

Tool results deserve their own cap. A database query that returns 5,000 rows is not a better answer than a bounded aggregation with a continuation path. For tools, define a response schema, a maximum row or byte count, a pagination contract, and a redaction step before results reach the packer. If an action needs additional evidence, make a second bounded call instead of carrying every prior result through the conversation.

This separation makes incident review practical. You can answer three useful questions from a trace: which instruction governed the response, which sources supported it, and which tool result changed its state. A single concatenated prompt cannot answer those questions reliably.

Evaluate the policy, not just the final answer

An answer score alone cannot tell you whether the failure came from retrieval, reranking, packing, or generation. Add a trace for every sampled request:

{
  "query_class": "troubleshooting",
  "evidence_budget": 2600,
  "candidate_count": 30,
  "reranked_count": 12,
  "accepted_chunk_ids": ["docs-18#4", "release-7#2"],
  "evidence_tokens": 1412,
  "answer_reserve": 900,
  "dropped": {"duplicate": 5, "budget": 4, "source_cap": 1}
}

Then evaluate representative questions by query class. For each, record whether the expected evidence was retrieved, admitted, cited, and actually supports the answer. This separates four common diagnoses:

Measure cost, latency, groundedness, and task success together. Lowering context tokens is not a win if correct answers disappear. Conversely, sending more text is not a win if it only increases cost and makes attribution worse.

Ship it with a narrow rollout

Use a shadow trace first: calculate the proposed pack while still sending the current one. Compare token counts and whether expected evidence would have been admitted. Next, run a small traffic cohort with a kill switch.

Test deliberately awkward cases: duplicated policy pages, an old and a new version, a query that needs two documents, a broad “list all” request, and a long conversation with a short current question. Include authorization changes in this suite. A perfect pack that leaks a cross-tenant passage is still a failure.

The goal is not the smallest possible prompt. It is the smallest sufficient evidence set for a task, with a predictable answer reserve and a record of why each passage was admitted.

FAQ

What is a RAG context budget?

It is the portion of an LLM request deliberately reserved for retrieved evidence after instructions, relevant conversation, the answer, and a safety margin have been allocated. It turns context assembly into an explicit policy instead of a fixed top_k setting.

How many chunks should a RAG system send to the model?

There is no universal number. Retrieve enough candidates for recall, then evaluate how many diverse passages each query class needs. A point lookup may need one authoritative section; an aggregation question may need several. Use a token ceiling and test against real tasks.

Should I increase top-k when RAG answers are wrong?

Not automatically. First find out whether the needed source was missing, retrieved but excluded, included but ignored, or cited incorrectly. Raising top_k helps only the first case and can make the other three worse.

How do I prevent duplicate chunks in RAG context?

Store stable source and section identities, detect overlap or semantic similarity among candidates, and cap passages per source. Preserve the strongest chunk and its provenance rather than sending several near-identical versions of the same fact.

Does a larger context window remove the need for RAG packing?

No. Larger windows can hold more material, but they do not make every passage relevant, authorized, current, or easy for a model to use. Packing still controls cost, attention, provenance, and answer space.

What should I log for a RAG context budget?

Log the query class, token allocations, candidate and accepted IDs, source versions, drop reasons, latency, and the final citations. Avoid storing sensitive raw text unless retention and access rules explicitly allow it.