DecisionRAG 架構深入剖析：以決策模型導覽文件，實現免檢索器檢索

Replacing Vector Databases with Decision-Based Tree Navigation in Production RAG

For the past three years, the industry standard for Retrieval-Augmented Generation (RAG) has been virtually dogmatic: chunk documents, compute dense vector embeddings, build an HNSW index, perform cosine similarity retrieval, and optionally throw an expensive cross-encoder reranker at the top-K candidates.

However, in high-stakes production architectures, this pipeline breaks in predictable, painful ways. Dense retrieval introduces an irrecoverable failure mode known as the retrieval capability asymmetry: if a bi-encoder projects a query and a passage into disjoint clusters due to semantic drift or lexical subtleties, your 70B generation model never even sees the context. You cannot recover what the first stage dropped.

Recent work on DecisionRAG (Retriever-Free Retrieval: Navigating Documents with Decision Models, arXiv:2610.35868) introduces a radical and practical architectural alternative: eliminating the vector database entirely in favor of hierarchical decision models.

The Production Failure Modes of Dense Vector Search

In enterprise systems dealing with structured technical knowledge, dense retrieval encounters four core operational bottlenecks:

Independent Evaluation Blindness: Bi-encoders calculate $sim(q, d_i)$ in absolute isolation. There is no relative context. A passage scoring $0.72$ might be optimal for Query A but utterly misleading for Query B.
The Re-indexing Tax: Updating your embedding model invalidates billions of vector records, demanding massive compute runs and zero-downtime blue/green index migrations.
Index Memory Footprint: Storing dense vectors with uncompressed HNSW graphs requires exorbitant RAM allocations inside clusters like Qdrant or Milvus.
Reranking Latency Penalty: Passing 50 candidates through an 8B cross-encoder sequentially destroys real-time latency budgets.

The Architectural Paradigm: Hierarchical Single-Pass Navigation

Instead of treating knowledge as an unorganized flat soup of chunks, DecisionRAG leverages the corpus's natural hierarchy: Topics $\to$ Sections $\to$ Subsections $\to$ Paragraphs.

The search behaves like an engineer browsing documentation:

The model starts at the root node containing broad taxonomy categories.
In a single forward pass, the model ingests all sibling branch summaries simultaneously.
Crucially, no tokens are generated. The engine measures the relative logits assigned to candidate branch identifiers in a shared context window.
The highest-scoring branches are expanded down to the leaf paragraph level.

Because the evaluation occurs across candidate branches inside the exact same attention context, the candidate relevance scores are genuinely comparable. It resolves the fundamental flaw of isolated vector dot-products.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

class DecisionTreeRetriever:
    def __init__(self, model_name: str):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(
            model_name, 
            torch_dtype=torch.bfloat16, 
            device_map="auto"
        )

    def evaluate_branches(self, query: str, branches: list[dict]) -> int:
        """
        Evaluates candidate branches in a shared context using a single forward pass.
        No token generation overhead.
        """
        options = "\n".join([
            f"[{idx}] {b['title']}: {b['summary']}" 
            for idx, b in enumerate(branches)
        ])

        prompt = (
            f"Query: {query}\n"
            f"Evaluate the most relevant branch to investigate:\n"
            f"{options}\n"
            f"Selected Branch: ["
        )

        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)

        with torch.no_grad():
            outputs = self.model(**inputs)
            next_token_logits = outputs.logits[0, -1, :]

        # Extract logits for branch index tokens
        branch_logits = []
        for idx in range(len(branches)):
            token_id = self.tokenizer.encode(str(idx), add_special_tokens=False)[-1]
            branch_logits.append(next_token_logits[token_id].item())

        return int(torch.tensor(branch_logits).argmax())

Benchmarks & Engineering Trade-offs

On MultiDoc2Dial, DecisionRAG demonstrates a +13.8 R@1 gain over an 8B dense reranker pipeline, and a +17.4 Hit@1 increase on Natural Questions. Across 2.68 million Wikipedia paragraphs, latency remains within 0.7 to 1.8 seconds per query.

Production Takeaways:

Drop the Vector Microservice: If your corpus has clear domain hierarchies (API documentation, legal contracts, codebase ASTs, ERP schemas), hierarchical trees serialized in fast key-value storage (e.g., LMDB, Redis, or DuckDB) completely replace vector search.
Zero Hallucination Retrieval: Because navigation is restricted to logits of explicit structural pointers, the retriever cannot hallucinate text during the retrieval step.
Deterministic Scaling: Tree depth scales logarithmically $\mathcal{O}(\log N)$. A corpus of millions of nodes requires only 3 to 4 forward passes.

As we advance autonomous multi-agent workflows, decoupling knowledge indexing from brittle geometric embeddings will be the definitive architectural shift for resilient AI infrastructure.

Canonical link: https://bohdanhordii.dev

Author: Bohdan Hordii | Website: bohdanhordii.dev | Contact: @bohdan_hordii