Data Substrate 與 RAG 向量資料庫的比較

This week's news about improved OCR capabilities for document images brought to mind a more fundamental question: how do we build systems that can reason about the data they process? When dealing with AI applications in regulated environments like Nigerian microfinance, the answer hinges on the distinction between a "data substrate" and vector databases used in RAG systems.

We're building a regtech tool for Nigerian microfinance banks that needs to analyze loan applications, detect fraud, and ensure compliance with evolving financial regulations. The data we work with is not just text - it's structured, versioned, and often comes with legal and audit implications. A vector database like Weaviate or Pinecone is useful for retrieving similar documents or answering questions based on unstructured content. But when we need to reason about the data - like verifying that a particular clause in a loan agreement was present in version 1.2 but not 1.3 - that's where a data substrate shines.

A data substrate is a versioned, citable, queryable layer that sits beneath the AI application. It's not a replacement for vector DBs, but a complement. For example, in our regtech tool, we use a data substrate to store and version all loan agreements, client data, and regulatory documents. Each version is immutable and can be queried for specific clauses, dates, or jurisdictions. This is crucial when auditors ask, "Was this term in the contract when the loan was approved?"

Here's how we model our data substrate using a PostgreSQL-based schema:

CREATE TABLE document_versions (
    id UUID PRIMARY KEY,
    document_hash TEXT NOT NULL,
    content JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL,
    version_number INT NOT NULL,
    document_type TEXT NOT NULL,
    metadata JSONB
);

Each document is stored as a JSONB blob, and we track its version number, creation timestamp, and metadata like the jurisdiction or regulation it relates to. This allows us to perform queries like:

SELECT * FROM document_versions
WHERE document_type = 'loan_agreement'
AND metadata->>'jurisdiction' = 'Nigeria'
AND metadata->>'regulation' = 'CBN 2023'
ORDER BY created_at DESC;

This approach gives us full traceability - a key requirement in regulated environments. A vector DB would struggle to provide this level of versioning and auditability without significant overhead.

But there are tradeoffs. A data substrate is not optimized for similarity searches or natural language queries. If we need to find all loan agreements that mention "collateral" in their content, we'd have to query the JSONB content field, which is less efficient than using a vector DB's similarity search. So, we use both: the data substrate for versioning and audit, and a vector DB for content-based retrieval.

What's next? We're exploring hybrid architectures that combine the best of both worlds - using the data substrate for versioned facts and the vector DB for semantic similarity. We're also experimenting with integrating this with on-device AI models, so that microfinance officers in remote parts of Nigeria can run compliance checks without relying on cloud infrastructure. How would you approach this problem if you were building for a similar regulated environment?