跳到正文
DEV Community· Le Trung Hieu·· 2 小時前AI 評分66

在 Amazon Bedrock 上構建正式環境 RAG 聊天機械人，實際會遇到甚麼？

What Actually Happens When You Build a Production RAG Chatbot on Amazon Bedrock

AI 導讀

作者以 NestJS、PostgreSQL、pgvector、混合搜尋（RRF）及 Claude on Amazon Bedrock，構建生產級文件問答系統。文章拆解文件匯入、分塊、向量與全文搜尋、提示詞快取和 API 流程，並示範如何用 RRF 合併兩路搜尋結果。文中亦指出重新索引時清除過期分塊等維護細節，並列出尚未實作的 Unicode 正規化、對話歷史改寫及引用驗證。

正文

How we built a document Q&A system with NestJS, PostgreSQL, pgvector, Hybrid Search (RRF), Prompt Caching, and Claude on Amazon Bedrock.
Building a demo chatbot is easy.

Building one that consistently answers correctly after weeks of document updates, changing requirements, and real user traffic is where things become interesting.

Most Retrieval-Augmented Generation (RAG) failures don't look like failures.

The API returns 200.

The model responds.

Infrastructure dashboards stay green.

Yet answer quality gradually gets worse.

You ask about error code E-4021 and receive documentation for E-4012.

A recently updated document never appears in search results.

Prompt caching is enabled, but costs don't go down.

Hybrid search is implemented, but retrieval quality barely improves.

Nothing crashes.

Nothing raises an alert.

The chatbot simply becomes less useful over time.

For teams involved in application software development, the real work begins after the first successful response. Retrieval quality, chunking strategy, embedding consistency, prompt caching, document lifecycle management, and hallucination control quickly become more important than model selection itself.

This article walks through a complete production-oriented RAG implementation built with:

  • Amazon Bedrock
  • Claude
  • Cohere Embed Multilingual v3
  • NestJS
  • PostgreSQL + pgvector
  • Hybrid Search using Reciprocal Rank Fusion (RRF)

More importantly, it explains why each component exists, what problem it solves, and which production issues it prevents.

1. What is RAG

RAG (Retrieval-Augmented Generation) does not ask the LLM directly. Before each question, the system:

  1. Finds a few internal document passages most relevant to the question.
  2. Pastes them into the prompt together with the question.
  3. Asks the LLM to answer based only on those passages, with citations.

The consequence of this design:

The problem "how does the LLM know my data" becomes a search problem. If retrieval is weak, the chatbot is weak, because no prompt can produce information that isn't in the context.

This distinction is easy to underestimate.

Many teams spend weeks experimenting with prompts while retrieval quality remains unchanged.

In reality, most enterprise application software development projects that incorporate AI succeed or fail based on retrieval architecture rather than prompt engineering.

That is why most of this article is about chunking documents and search, not about prompts.

Compared with the alternatives:

  • Stuffing all documents into the prompt: only works for a very small corpus (a few dozen pages). Beyond that you hit the context limit and pay for the tokens on every turn.
  • Fine-tuning: teaches style, not facts. A fine-tuned model still makes things up, just in your company's voice, which makes it harder to spot. Every data change means retraining.
  • RAG: updating knowledge only requires re-ingesting, and answers can cite their sources. This is the default choice for business chatbots.

2. Overall architecture

A RAG system has three stages:

  • Indexing (offline, re-run whenever documents change): read documents → chunk → generate embeddings → store in an index.
  • Retrieval (online, per question): embed the question, find the most relevant chunks.
  • Generation (online, per question): put the chunks in the prompt, the LLM writes the answer.

The three stages of RAG: documents are indexed ahead of time, then each question goes through retrieval and then generation

 ══ OFFLINE ══════════════════════════════════════════════════════════
   data/in/*.md|txt|json
        │
        ▼
   LoaderService ──► ChunkerService ──► Bedrock (Cohere Embed) ──► Postgres
   read files        2-level split +    input_type =               chunks table
                     header injection   search_document            (vector + tsvector)

 ══ ONLINE ═══════════════════════════════════════════════════════════
   POST /api/chat { question }
        │
        ▼
   SearchService.hybridSearch
        ├─ embedQuery (Cohere, input_type = search_query)
        ├─ DenseSearch   (HNSW index)  ─┐  run in parallel
        ├─ KeywordSearch (GIN index)   ─┘
        └─ reciprocalRankFusion ──► top 5 chunks
        │
        ▼
   PromptBuilder ── [system + few-shot: CACHED] + [context + question: not cached]
        │
        ▼
   Bedrock (Claude, InvokeModel)
        │
        ▼
   { answer, citations, usage }

Folder layout, one folder per stage:

src/
├── config/     all config, read from env
├── database/   schema.sql + pg.Pool wrapper (no ORM)
├── llm/        LlmService interface + Bedrock implementation
├── ingest/     loader → chunker → repository → ingest.service
├── search/     dense · keyword · rrf · search.service
├── prompt/     system prompt · few-shot · prompt builder
└── chat/       wires the three steps into one RAG loop + HTTP API
scripts/        db-init · ingest · search (CLI)
test/           unit tests for chunker, rrf, prompt builder

This architecture may look surprisingly simple.

That is intentional.

Every stage has a single responsibility and can be tested independently.

When retrieval quality drops, we can inspect retrieval.

When generation quality drops, we can inspect prompts.

When ingest breaks, we can inspect indexing.

The separation becomes extremely valuable once the system grows beyond a proof of concept.

3. Following the data: the two flows as they run in the code

The diagram in section 2 shows the blocks. This section slows down: it takes one concrete document and one concrete question and follows them through each function, each SQL query, and each Bedrock call. Once you have read this section, the code in the later steps is just detail.

The sample data used throughout is data/in/warranty.md:

# Warranty Policy

## Warranty period
All products are covered by a warranty of 12 months from the date of purchase.

## Returns
Products with a manufacturing defect can be exchanged within the first 7 days.

3.1 The ingest flow: from raw files to the index

This flow runs offline via npm run ingest, not over HTTP. Re-run it whenever documents change.

npm run ingest [-- <folder>]
   │
   ▼
scripts/ingest.ts
   │  NestFactory.createApplicationContext(AppModule)   (no HTTP port opened)
   ▼
IngestService.ingestDirectory(dir = INGEST_DIR)
   │
   ▼
[1] LoaderService.loadDirectory ─────────► RawDocument[]
   │    recursive scan; .md/.txt = 1 document; .json = n documents; other extensions skipped
   │
   │    for EACH RawDocument:
   ▼
[2] ChunkerService.chunk ────────────────► Chunk[]
   │    split by heading, split long sections (1200 chars, 120 overlap), prepend heading
   ▼
[3] BedrockService.embedDocuments ───────► one 1024-dim vector per chunk
   │    up to 96 texts per call, Cohere input_type = search_document
   ▼
[4] ChunkRepository.upsert (per chunk) ──► chunks table
   │    INSERT ... ON CONFLICT (chunk_id) DO UPDATE
   │    Postgres computes search_vector and updates HNSW + GIN indexes
   ▼
[5] ChunkRepository.deleteStale ─────────► delete leftover chunks of this document
   │
   ▼
IngestReport { documents, chunks, deleted }  →  printed to console, pool closed

[1] Read files. LoaderService walks the folder recursively. A markdown file becomes one RawDocument:

{
  "source": "data/in/warranty.md",
  "title": "Warranty Policy",
  "text": "# Warranty Policy\n\n## Warranty period\nAll products are covered..."
}

source is the file path, and title comes from the first H1 (or the file name if there is none). For .json files, each element with a text field becomes its own document.

[2] Chunk. splitByHeading reads line by line and starts a new section whenever a line begins with #. The sample file produces 3 sections:

section A  heading = "Warranty Policy"   text = ""                  → no chunk
section B  heading = "Warranty period"   text = "All products are..."
section C  heading = "Returns"           text = "Products with a..."

Section A has a heading but no body, so it is dropped. Sections B and C are shorter than 1200 characters, so they are kept whole. A longer section would be split further by splitLongText: paragraph boundaries first, then sentences, with 120 characters of overlap between consecutive pieces.

After prepending the heading (header injection), we get 2 chunks:

[
  {
    "chunkId": "data/in/warranty.md#0",
    "title": "Warranty period",
    "content": "Warranty period\n\nAll products are covered by a warranty of 12 months from the date of purchase.",
    "metadata": { "heading": "Warranty period" }
  },
  {
    "chunkId": "data/in/warranty.md#1",
    "title": "Returns",
    "content": "Returns\n\nProducts with a manufacturing defect can be exchanged within the first 7 days.",
    "metadata": { "heading": "Returns" }
  }
]

chunkId has the form <source>#<position>. It is the key that lets the next ingest update the existing row instead of adding a new one.

[3] Embed. Both content strings go to Cohere in a single InvokeModel call:

{ "texts": ["Warranty period\n\nAll products are...", "Returns\n\nProducts with a..."],
  "input_type": "search_document", "truncate": "END" }

The result is 2 vectors of 1024 floats each. Because the heading is part of content, chunk #0's vector carries the meaning "warranty period", not just the sentence about 12 months.

[4] Store. Each chunk is one INSERT ... ON CONFLICT (chunk_id) DO UPDATE. The application only writes content and embedding. Postgres does the rest:

  • The search_vector column is recomputed from title + content with to_tsvector('simple', ...). Chunk #0 yields the tokens warranty, period, all, products, are, covered, by, a, of, 12, months, from, the, date, purchase.
  • The HNSW index (on embedding) and the GIN index (on search_vector) are updated in the same statement.

[5] Clean up leftovers. deleteStale('data/in/warranty.md', ['...#0', '...#1']) deletes every row of this file whose chunk_id is not in the list just written. On the first run there is nothing to delete. If the file later shrinks from 5 chunks to 2, rows #2, #3 and #4 are deleted here.

At the end, the console prints a summary: number of documents, chunks, and stale chunks deleted.

Three things worth knowing about this flow:

  • Every run re-embeds everything, even unchanged files. Fine for a small corpus; for a large one, store a content hash in metadata and skip chunks whose hash hasn't changed.
  • source is the path exactly as scanned. Running npm run ingest (giving data/in/...) and later npm run ingest -- /absolute/path produces a different source, i.e. a duplicate set of chunks. Always use the same form of path.
  • A file that yields no chunks is skipped, and its old chunks are not cleaned up. To remove a document from the index entirely, use ChunkRepository.deleteBySource.

3.2 The question flow: from question to answer

This flow runs online, once per request. One question costs 2 Bedrock calls (1 embed, 1 generate) and 2 SQL queries run in parallel.

Client
   │  POST /api/chat  { "question": "How long is the product warranty?" }
   ▼
[1] ValidationPipe + AskDto ─────────────► 400 if empty, > 2000 chars, or unknown fields
   │
   ▼
ChatController.ask  →  ChatService.ask
   │
   ▼
[2] SearchService.hybridSearch(question)
   │
   ├─ [2a] BedrockService.embedQuery ────► question vector (Cohere, input_type = search_query)
   │
   ├─ [2b] in parallel (Promise.all):
   │      ├─ DenseSearch   : ORDER BY embedding <=> $vec LIMIT 50   → HNSW index
   │      └─ KeywordSearch : search_vector @@ websearch_to_tsquery  → GIN index
   │
   └─ [2c] reciprocalRankFusion(k = 60) ─► top 5 chunks, with denseRank / keywordRank
   │
   ▼
[3] PromptBuilder.build ─────────────────► { system: [...], user: [...] }
   │      system = SYSTEM_PROMPT + few-shot                    (cached)
   │      user   = <context>[1]..[5]</context> + <question>    (not cached)
   ▼
[4] BedrockService.generate ─────────────► InvokeModel Claude
   │      cache_control on the few-shot block; retry on throttling
   ▼
[5] ChatService assembles the result
   │      answer = all "text" blocks joined
   │      citations = the chunks that were sent, numbered to match [n]
   │      usage = input/output tokens + cache write/read tokens
   ▼
Client receives { answer, citations, usage }

[1] Guard the input. ValidationPipe runs with whitelist and forbidNonWhitelisted, so the body may contain only the question field. AskDto requires question to be a non-empty string of at most 2000 characters. Invalid requests get a 400 immediately, before a single cent is spent on Bedrock.

[2a] Embed the question. The question goes to Cohere with input_type: "search_query". This mirrors step [3] of the ingest flow: documents use search_document, questions use search_query, same model. Get either one wrong and question vectors and document vectors no longer live on the same "map".

[2b] Two search branches run at the same time.

The dense branch compares the question vector with every chunk vector by cosine; thanks to HNSW it does not scan the whole table. With our 2-chunk sample corpus:

dense:   1. ...#0  "Warranty period"   highest cosine
         2. ...#1  "Returns"

The keyword branch passes the raw question to websearch_to_tsquery('simple', ...):

'how' & 'long' & 'is' & 'the' & 'product' & 'warranty'

This function joins words with AND. Chunk #0 contains none of how, long, is, and has products rather than product (the simple dictionary does not stem), so the keyword branch returns 0 results for this question. That is the real behavior of the code, and the reason the dense branch matters: for long natural-language questions the keyword branch usually stays silent. It shines on short queries containing codes or names, like E-4021 or warranty 12 months.

[2c] Merge with RRF. Each chunk gets 1 / (60 + rank) from every branch it appears in:

...#0  rrf = 1/61 = 0.01639   denseRank = 1   keywordRank = null
...#1  rrf = 1/62 = 0.01613   denseRank = 2   keywordRank = null

If the query were warranty 12 months, chunk #0 would appear in both branches, score 1/61 + 1/61 = 0.03279, and pull far ahead of chunks found by only one branch. The list is cut to the top 5 (TOP_K).

[3] Build the prompt. PromptBuilder numbers the chunks from 1 and wraps them in tags (the repository uses Vietnamese prompt text and tag names; English equivalents are shown throughout this article, see section 12):

<context>
[1] Warranty period
Warranty period

All products are covered by a warranty of 12 months from the date of purchase.

[2] Returns
Returns

Products with a manufacturing defect can be exchanged within the first 7 days.
</context>
<question>How long is the product warranty?</question>

The heading appears twice because title is printed first while content already contains the heading from header injection. It's harmless, but you can drop the title line when rendering to save tokens.

If neither branch finds anything, the context is (no relevant documents found) and the LLM is still called. The system prompt and few-shot example #2 teach the model to say it could not find the information instead of inventing an answer.

[4] Call Claude. The body sent via InvokeModel:

{
  "anthropic_version": "bedrock-2023-05-31",
  "max_tokens": 2000,
  "system": [
    { "type": "text", "text": "You are a Q&A assistant that answers from the provided documents..." },
    { "type": "text", "text": "Examples of good answers: ...",
      "cache_control": { "type": "ephemeral" } }
  ],
  "messages": [
    { "role": "user", "content": [
      { "type": "text", "text": "<context>\n[1] Warranty period\n...\n</context>" },
      { "type": "text", "text": "<question>How long is the product warranty?</question>" }
    ] }
  ]
}

Everything before cache_control (system prompt + few-shot) is identical on every request, so from the second request on it is read from cache. messages changes with every question and is always billed at full price. If Bedrock returns ThrottlingException or ServiceUnavailableException, the request is retried after 500ms and then 1s; any other error is thrown immediately.

[5] Return the result. Claude returns a content array. The code keeps only type: "text" blocks and joins them, ignoring other block types. citations is the list of chunks put into the prompt, numbered to match [n] in the context, so the client can turn [1] in the answer into a link to source:

{
  "answer": "All products are covered by a 12-month warranty from the date of purchase [1].",
  "citations": [
    { "index": 1, "chunkId": "data/in/warranty.md#0", "source": "data/in/warranty.md", "title": "Warranty period" },
    { "index": 2, "chunkId": "data/in/warranty.md#1", "source": "data/in/warranty.md", "title": "Returns" }
  ],
  "usage": { "inputTokens": 180, "outputTokens": 25, "cacheWriteTokens": 0, "cacheReadTokens": 650 }
}

(The usage values above are illustrative.) Note that citations contains every chunk that was sent, not only the ones the model actually cited. The client should filter by the [n] numbers that appear in answer.

3.3 What makes answers accurate, and at which step

Accuracy doesn't come from one place. It accumulates across both flows:

  • Ingest [2], header injection: chunks carry their section heading, so questions phrased like the heading still match the body text.
  • Ingest [2], split by heading first, by length second: each chunk is about one idea, so its vector isn't "averaged out" across several topics.
  • Ingest [3] + Question [2a], correct input_type: documents and questions are projected into the same vector space.
  • Ingest [5], cleaning up leftovers: content deleted from a document can no longer leak into answers.
  • Question [2b], two branches + over-fetching 50: the dense branch catches paraphrases, the keyword branch catches codes and names; over-fetching keeps chunks that do well in both branches from falling off the list.
  • Question [2c], RRF: chunks found by both branches rise to the top.
  • Question [3], system prompt rules + a refusal few-shot: the model answers only from context, cites [n], and knows how to say "not found".

What these two flows don't do yet, covered in the "Upgrade path" section at the end: Unicode NFC normalization for both documents and questions, rewriting questions using conversation history (the API is currently stateless), a confidence threshold to skip the LLM when results are too weak, and checking in code that the [n] the model cites actually exist.

4. Tech stack and why

One design goal guided every technology choice: keep the architecture understandable.

Every component should be replaceable without rewriting the rest of the system.

  • NestJS 10 (TypeScript): modules + dependency injection make each layer replaceable on its own. Switching LLM providers is a one-line useClass change.
  • Amazon Bedrock: one API and one AWS bill for both embeddings and generation. Credentials follow your organization's existing IAM.
  • Cohere Embed Multilingual v3 (cohere.embed-multilingual-v3, 1024 dimensions): multilingual, which matters when documents mix a local language with English UI labels.
  • Claude on Bedrock: generates answers and supports prompt caching.
  • PostgreSQL 16 + pgvector: vector search (HNSW) and full-text search (GIN) in the same database. No extra service, no data sync between two systems.
  • Plain pg, no ORM: SQL lives next to the module that uses it, so reading a query tells you which index it uses.
  • @aws-sdk/client-bedrock-runtime: calls InvokeModel.
  • Jest, ESLint/Prettier, multi-stage Docker, GitHub Actions.

That's the entire runtime dependency list:

"dependencies": {
  "@aws-sdk/client-bedrock-runtime": "^3.696.0",
  "@nestjs/common": "^10.4.0",
  "@nestjs/core": "^10.4.0",
  "@nestjs/platform-express": "^10.4.0",
  "class-transformer": "^0.5.1",
  "class-validator": "^0.14.1",
  "dotenv": "^16.4.5",
  "pg": "^8.12.0",
  "reflect-metadata": "^0.2.2",
  "rxjs": "^7.8.1"
}

5. Prerequisites

5.1 Local environment

  • Node.js 20+
  • Docker + Docker Compose
  • An AWS account

5.2 Enable models on Amazon Bedrock

  1. Open the Amazon Bedrock console and pick your region (this article uses ap-northeast-1).
  2. Make sure your account has access to two models: Cohere Embed Multilingual v3 and a Claude model. The first time you use an Anthropic model, AWS may ask you to fill in a use-case form.
  3. Open the Model catalog and note the exact model ID for Claude. Bedrock model IDs have the anthropic. prefix, for example anthropic.claude-opus-5-5. Some regions require calling through a cross-region inference profile; in that case use the profile ID the console shows.

5.3 Minimal IAM policy

The application only needs bedrock:InvokeModel:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["bedrock:InvokeModel"],
      "Resource": [
        "arn:aws:bedrock:*::foundation-model/*",
        "arn:aws:bedrock:*:*:inference-profile/*"
      ]
    }
  ]
}

In production, narrow Resource down to the exact ARNs of the two models you use.

About credentials: static access keys work, but it's better to leave them empty and let the AWS SDK pick up AWS_PROFILE, SSO, or the EC2/ECS IAM role. The code below supports both.

6. Step 1: Create the project 

npm i -g @nestjs/cli
nest new rag-chatbot --package-manager npm --skip-git
cd rag-chatbot

npm i @aws-sdk/client-bedrock-runtime pg dotenv class-validator class-transformer
npm i -D @types/pg ts-node

Add these scripts to package.json:

"scripts": {
  "build": "nest build",
  "start:dev": "nest start --watch",
  "start:prod": "node dist/main",
  "db:init": "ts-node scripts/db-init.ts",
  "ingest": "ts-node scripts/ingest.ts",
  "search": "ts-node scripts/search.ts",
  "test": "jest",
  "lint": "eslint \"{src,scripts,test}/**/*.ts\""
}

7. Step 2: Config in one place

Every setting is read from env, with sensible defaults so it runs out of the box. No per-project profile files.

.env.example:

# ── App
PORT=8000

# ── PostgreSQL + pgvector
DB_HOST=127.0.0.1
DB_PORT=5432
DB_NAME=ragdb
DB_USER=postgres
DB_PASSWORD=postgres
# Must match the embedding model's dimensions and the vector(N) column in the schema.
# cohere.embed-multilingual-v3 → 1024. Changing it = re-run db:init + re-embed everything.
EMBEDDING_DIMENSIONS=1024

# ── AWS Bedrock
AWS_REGION=ap-northeast-1
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_SESSION_TOKEN=
# Leave the keys above empty to let the SDK use AWS_PROFILE / SSO / instance role.

BEDROCK_EMBEDDING_MODEL_ID=cohere.embed-multilingual-v3
BEDROCK_GENERATION_MODEL_ID=anthropic.claude-opus-5-5

# ── Ingest / chunking
INGEST_DIR=./data/in
CHUNK_MAX_CHARS=1200
CHUNK_OVERLAP_CHARS=120
EMBED_BATCH_SIZE=96

# ── Search
TOP_K=5            # number of chunks finally placed in the prompt
CANDIDATE_LIMIT=50 # candidates fetched by EACH branch before fusion
RRF_K=60           # RRF constant k, 60 is the value from the original paper

# ── Generation
MAX_TOKENS=2000
ENABLE_PROMPT_CACHE=true

src/config/config.ts (abridged; the full version has an AppConfig interface documenting each field):

const num = (v: string | undefined, fallback: number) =>
  v === undefined || v === '' ? fallback : Number(v);

export function loadConfig(): AppConfig {
  return {
    port: num(process.env.PORT, 8000),
    database: {
      host: process.env.DB_HOST || '127.0.0.1',
      port: num(process.env.DB_PORT, 5432),
      name: process.env.DB_NAME || 'ragdb',
      user: process.env.DB_USER || 'postgres',
      password: process.env.DB_PASSWORD || 'postgres',
      embeddingDimensions: num(process.env.EMBEDDING_DIMENSIONS, 1024),
    },
    bedrock: {
      region: process.env.AWS_REGION || 'ap-northeast-1',
      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
      sessionToken: process.env.AWS_SESSION_TOKEN,
      embeddingModelId: process.env.BEDROCK_EMBEDDING_MODEL_ID || 'cohere.embed-multilingual-v3',
      generationModelId: process.env.BEDROCK_GENERATION_MODEL_ID || 'anthropic.claude-opus-5-5',
    },
    ingest: {
      dir: process.env.INGEST_DIR || './data/in',
      chunkMaxChars: num(process.env.CHUNK_MAX_CHARS, 1200),
      chunkOverlapChars: num(process.env.CHUNK_OVERLAP_CHARS, 120),
      embedBatchSize: num(process.env.EMBED_BATCH_SIZE, 96),
    },
    search: {
      topK: num(process.env.TOP_K, 5),
      candidateLimit: num(process.env.CANDIDATE_LIMIT, 50),
      rrfK: num(process.env.RRF_K, 60),
    },
    generation: {
      maxTokens: num(process.env.MAX_TOKENS, 2000),
      enableCache: process.env.ENABLE_PROMPT_CACHE !== 'false',
    },
  };
}

export const CONFIG = Symbol('CONFIG');

Register CONFIG as a global provider in AppConfigModule ({ provide: CONFIG, useFactory: loadConfig }) so every service can inject it with @Inject(CONFIG).

8. Step 3: PostgreSQL + pgvector

8.1 Docker Compose 

name: rag-core

services:
  postgres:
    image: pgvector/pgvector:pg16   # pgvector 0.7+, needed for the HNSW index
    container_name: rag-postgres
    environment:
      POSTGRES_USER: ${DB_USER:-postgres}
      POSTGRES_PASSWORD: ${DB_PASSWORD:-postgres}
      POSTGRES_DB: ${DB_NAME:-ragdb}
    ports:
      - "${DB_PORT:-5432}:5432"
    volumes:
      - pgvector_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-postgres} -d ${DB_NAME:-ragdb}"]
      interval: 5s
      timeout: 5s
      retries: 10

volumes:
  pgvector_data:

name: rag-core is there for a reason: without it, Compose uses the folder name as the project name, and two checkouts with the same folder name on one machine will silently recreate each other's containers.

8.2 Schema: one table, three indexes

src/database/schema.sql:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE IF NOT EXISTS chunks (
  id           BIGSERIAL PRIMARY KEY,
  -- Logical key "<source>#<index>": re-ingesting the same file upserts the same row.
  chunk_id     TEXT NOT NULL UNIQUE,
  source       TEXT NOT NULL,
  chunk_index  INT  NOT NULL,
  title        TEXT,
  content      TEXT NOT NULL,
  metadata     JSONB NOT NULL DEFAULT '{}'::jsonb,
  embedding    vector(1024),

  -- Auto-generated full-text column, 'simple' dictionary.
  search_vector tsvector GENERATED ALWAYS AS (
    to_tsvector('simple', coalesce(title, '') || ' ' || content)
  ) STORED,

  created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Vector search: HNSW + cosine
CREATE INDEX IF NOT EXISTS chunks_embedding_idx
  ON chunks USING hnsw (embedding vector_cosine_ops);

-- Full-text search
CREATE INDEX IF NOT EXISTS chunks_fts_idx
  ON chunks USING gin (search_vector);

CREATE INDEX IF NOT EXISTS chunks_source_idx ON chunks (source);

Three decisions worth noting:

  • chunk_id is a unique logical key. Thanks to it, running ingest a second time updates rows (upsert) instead of duplicating them.
  • search_vector is a generated column. Postgres updates it whenever content changes, so application code never has to keep it in sync.
  • The 'simple' dictionary instead of 'english'. simple does no stemming and removes no language-specific stopwords. For languages like Vietnamese, the english config would chop word endings in nonsensical ways. simple is less sophisticated but never damages the data, and it keeps error codes, product names, and version numbers intact. Those are exactly what the keyword branch needs to catch.

8.3 Schema setup script

scripts/db-init.ts reads the SQL file, swaps in the vector dimension from EMBEDDING_DIMENSIONS, and runs it. Safe to run any number of times (every statement is IF NOT EXISTS):

const sql = await fs.readFile(path.join(__dirname, '../src/database/schema.sql'), 'utf8');
await client.connect();
await client.query(sql.replace('vector(1024)', `vector(${config.database.embeddingDimensions})`));
await client.end();

8.4 Database service: a thin wrapper around pg.Pool 

@Injectable()
export class DatabaseService implements OnModuleDestroy {
  private readonly pool: Pool;

  constructor(@Inject(CONFIG) config: AppConfig) {
    this.pool = new Pool({
      host: config.database.host,
      port: config.database.port,
      database: config.database.name,
      user: config.database.user,
      password: config.database.password,
    });
  }

  async query<T extends QueryResultRow>(sql: string, params: unknown[] = []): Promise<T[]> {
    const result = await this.pool.query<T>(sql, params);
    return result.rows;
  }

  /** pgvector accepts a vector as the literal '[1,2,3]' cast to ::vector. */
  static toVector(values: number[]): string {
    return `[${values.join(',')}]`;
  }

  async onModuleDestroy(): Promise<void> {
    await this.pool.end();
  }
}

9. Step 4: The LLM layer, calling Bedrock

9.1 Interface first, implementation second

The rest of the codebase only knows this interface, not Bedrock:

export const LLM = Symbol('LLM');

export interface PromptBlock {
  text: string;
  /** true = block belongs to the static prefix and gets a cache_control marker. */
  cacheable?: boolean;
}

export interface GenerateRequest {
  system: PromptBlock[]; // persona, rules, few-shot: the cached part
  user: PromptBlock[];   // retrieved context + question: changes every turn
  maxTokens?: number;
}

export interface Usage {
  inputTokens: number;
  outputTokens: number;
  cacheWriteTokens: number;
  cacheReadTokens: number;
}

export interface LlmService {
  embedDocuments(texts: string[]): Promise<number[][]>;
  embedQuery(text: string): Promise<number[]>;
  generate(request: GenerateRequest): Promise<{ text: string; usage: Usage }>;
}

// src/llm/llm.module.ts: switching providers = changing this one useClass line
@Global()
@Module({
  providers: [{ provide: LLM, useClass: BedrockService }],
  exports: [LLM],
})
export class LlmModule {}

9.2 Client and retries 

@Injectable()
export class BedrockService implements LlmService {
  private readonly client: BedrockRuntimeClient;

  constructor(@Inject(CONFIG) private readonly config: AppConfig) {
    const { region, accessKeyId, secretAccessKey, sessionToken } = config.bedrock;
    this.client = new BedrockRuntimeClient({
      region,
      // Don't pass credentials when there are no static keys,
      // so the SDK uses its own chain (AWS_PROFILE, SSO, instance role).
      ...(accessKeyId && secretAccessKey
        ? { credentials: { accessKeyId, secretAccessKey, sessionToken } }
        : {}),
    });
  }

  private async invoke(modelId: string, body: unknown): Promise<Record<string, any>> {
    let lastError: unknown;

    for (let attempt = 1; attempt <= 3; attempt++) {
      try {
        const response = await this.client.send(
          new InvokeModelCommand({
            modelId,
            contentType: 'application/json',
            accept: 'application/json',
            body: JSON.stringify(body),
          }),
        );
        return JSON.parse(new TextDecoder().decode(response.body));
      } catch (err) {
        lastError = err;
        const name = (err as { name?: string }).name ?? '';
        const retryable = name === 'ThrottlingException' || name === 'ServiceUnavailableException';
        if (!retryable || attempt === 3) break;
        await new Promise((r) => setTimeout(r, 2 ** attempt * 250)); // 500ms, 1s
      }
    }
    throw lastError;
  }

Only transient errors (throttling, service unavailable) are retried, with exponential backoff. A ValidationException or AccessDeniedException won't change no matter how many times you retry, so those are thrown immediately.

9.3 Embeddings with Cohere: don't forget input_type 

  private async embed(
    texts: string[],
    inputType: 'search_document' | 'search_query',
  ): Promise<number[][]> {
    const out: number[][] = [];
    const batchSize = Math.min(this.config.ingest.embedBatchSize, 96); // Cohere: max 96 texts per call

    for (let i = 0; i < texts.length; i += batchSize) {
      const body = await this.invoke(this.config.bedrock.embeddingModelId, {
        texts: texts.slice(i, i + batchSize),
        input_type: inputType,
        truncate: 'END',
      });
      out.push(...(body.embeddings as number[][]));
    }
    return out;
  }

  async embedDocuments(texts: string[]) {
    return texts.length ? this.embed(texts, 'search_document') : [];
  }

  async embedQuery(text: string) {
    const [embedding] = await this.embed([text], 'search_query');
    return embedding;
  }

input_type directly affects results. Cohere only projects documents and queries into the same space when each side declares its role correctly: documents are search_document, questions are search_query. Using the same value for both still runs and reports no error; recall just drops a few percent, and hardly anyone notices.

9.4 Generating answers with Claude + cache markers

The body sent to Claude via InvokeModel follows the Anthropic Messages format, with anthropic_version: 'bedrock-2023-05-31':

type TextBlock = { type: 'text'; text: string; cache_control?: { type: 'ephemeral' } };

  /** Put cache_control on the LAST block of the static prefix. */
  private toBlocks(blocks: PromptBlock[]): TextBlock[] {
    const out: TextBlock[] = blocks
      .filter((b) => b.text.trim())
      .map((b) => ({ type: 'text' as const, text: b.text }));

    if (!this.config.generation.enableCache) return out;

    const lastCacheable = blocks.reduce(
      (acc, b, i) => (b.cacheable && b.text.trim() ? i : acc),
      -1,
    );
    if (lastCacheable >= 0 && out[lastCacheable]) {
      out[lastCacheable].cache_control = { type: 'ephemeral' };
    }
    return out;
  }

  async generate(request: GenerateRequest) {
    const body = await this.invoke(this.config.bedrock.generationModelId, {
      anthropic_version: 'bedrock-2023-05-31',
      max_tokens: request.maxTokens ?? this.config.generation.maxTokens,
      system: this.toBlocks(request.system),
      messages: [{ role: 'user', content: this.toBlocks(request.user) }],
    });

    // Keep only 'text' blocks. Newer Claude models may return other block
    // types (e.g. thinking); without this filter they leak into the answer.
    const text = ((body.content ?? []) as Array<{ type: string; text?: string }>)
      .filter((block) => block.type === 'text')
      .map((block) => block.text ?? '')
      .join('')
      .trim();

    const usage = body.usage ?? {};
    return {
      text,
      usage: {
        inputTokens: usage.input_tokens ?? 0,
        outputTokens: usage.output_tokens ?? 0,
        cacheWriteTokens: usage.cache_creation_input_tokens ?? 0,
        cacheReadTokens: usage.cache_read_input_tokens ?? 0,
      },
    };
  }
}

Why the cache marker goes on the last block of the static part: see Step 7.

10. Step 5: The ingest pipeline

The pipeline has four classes, each with one job:

LoaderService  ──►  ChunkerService  ──►  LlmService.embedDocuments  ──►  ChunkRepository
(read sources)      (chunking)            (batches of 96)                 (upsert + clean up)
                         ▲
                  IngestService only orchestrates, no logic of its own

10.1 Loader: the only place that knows where data comes from 

export interface RawDocument {
  source: string;   // file path, URL, CMS id...
  title?: string;
  text: string;
  metadata?: Record<string, unknown>;
}

LoaderService scans the folder recursively:

  • .md / .txt: reads the whole file. Uses the first H1 as the title, or the file name if there is none.
  • .json: an array of objects with a text field (optional source, title, metadata), handy for data exported from a CMS or database. 
async loadTextFile(file: string): Promise<RawDocument> {
  const text = await fs.readFile(file, 'utf8');
  return {
    source: file,
    title: /^#\s+(.+)$/m.exec(text)?.[1]?.trim() ?? path.basename(file),
    text,
  };
}

To add S3, Confluence, Notion and so on, add one method that returns RawDocument[]. Chunking and embedding stay the same.

10.2 Chunker: two-level splitting + header injection

This step has a big impact on quality but is often done carelessly.

  • Chunks too large: one chunk covers three topics, its embedding is "averaged out" and represents none of them.
  • Fixed character cuts: sentences are cut mid-way, leaving meaningless fragments.
  • Chunks lose context on their own: the passage "If the code cannot be read, enter the product code manually." contains no word about "inventory check" or "scanner", so a real user question will match it only weakly.

The codebase's strategy:

  1. Level 1: split by markdown heading. Headings are natural semantic boundaries, so use them first.
  2. Level 2: any section still longer than CHUNK_MAX_CHARS is split further by paragraph (\n\n), then by sentence, with an overlap between consecutive chunks. A hard cut only happens when a single sentence is longer than the limit.
  3. Header injection: prepend the section heading to the chunk content before embedding. Each chunk is embedded and full-text indexed together with its context. 
@Injectable()
export class ChunkerService {
  constructor(@Inject(CONFIG) private readonly config: AppConfig) {}

  chunk(doc: RawDocument): Chunk[] {
    const { chunkMaxChars, chunkOverlapChars } = this.config.ingest;
    const pieces: Array<{ heading?: string; text: string }> = [];

    for (const section of this.splitByHeading(doc.text)) {
      for (const part of this.splitLongText(section.text, chunkMaxChars, chunkOverlapChars)) {
        pieces.push({ heading: section.heading, text: part });
      }
    }

    return pieces.map((piece, index) => ({
      chunkId: `${doc.source}#${index}`,
      source: doc.source,
      chunkIndex: index,
      title: piece.heading ?? doc.title,
      // Header injection: the heading goes into the embedding and FTS with the content.
      content: piece.heading ? `${piece.heading}\n\n${piece.text}` : piece.text,
      metadata: { ...(doc.metadata ?? {}), heading: piece.heading ?? null },
    }));
  }

  private splitByHeading(text: string) {
    const sections: Array<{ heading?: string; text: string[] }> = [];
    let current: { heading?: string; text: string[] } = { text: [] };

    for (const line of text.split('\n')) {
      const heading = /^(#{1,6})\s+(.+)$/.exec(line);
      if (heading) {
        if (current.text.join('').trim() || current.heading) sections.push(current);
        current = { heading: heading[2].trim(), text: [] };
      } else {
        current.text.push(line);
      }
    }
    sections.push(current);

    return sections
      .map((s) => ({ heading: s.heading, text: s.text.join('\n').trim() }))
      .filter((s) => s.text || s.heading);
  }

  private splitLongText(text: string, maxChars: number, overlapChars: number): string[] {
    if (text.length <= maxChars) return text ? [text] : [];

    // Prefer paragraph boundaries, then sentences (Japanese punctuation included).
    const units = text
      .split(/\n{2,}/)
      .flatMap((p) => (p.length <= maxChars ? [p] : p.split(/(?<=[.!?。？！])\s+/)))
      .filter((u) => u.trim());

    const out: string[] = [];
    let buffer = '';

    const flush = () => {
      if (!buffer.trim()) return;
      out.push(buffer.trim());
      buffer = overlapChars > 0 ? buffer.slice(-overlapChars) : ''; // overlap
    };

    for (const unit of units) {
      if (buffer && buffer.length + unit.length + 2 > maxChars) flush();

      if (unit.length > maxChars) {
        // The smallest unit is still too long, no boundary left to respect.
        for (let i = 0; i < unit.length; i += maxChars - overlapChars) {
          out.push(unit.slice(i, i + maxChars).trim());
        }
        buffer = '';
        continue;
      }
      buffer = buffer ? `${buffer}\n\n${unit}` : unit;
    }

    if (buffer.trim()) out.push(buffer.trim());
    return out.filter(Boolean);
  }
}

If you can make only one chunking improvement, make it header injection: it costs almost nothing (string concatenation) and improves both the vector branch and the keyword branch.

A note on size: CHUNK_MAX_CHARS=1200 counts characters. For the same content, Vietnamese takes roughly 1.5x as many tokens as English, and Japanese roughly 3x. For Japanese documents, lower this limit so chunks stay within the embedding model's limit (anything beyond it is silently cut by truncate: 'END').

10.3 Repository: upsert and clean up old chunks 

async upsert(chunk: Chunk, embedding: number[]): Promise<void> {
  await this.db.query(
    `
    INSERT INTO chunks (chunk_id, source, chunk_index, title, content, metadata, embedding)
    VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7::vector)
    ON CONFLICT (chunk_id) DO UPDATE SET
      source = EXCLUDED.source, chunk_index = EXCLUDED.chunk_index,
      title = EXCLUDED.title, content = EXCLUDED.content,
      metadata = EXCLUDED.metadata, embedding = EXCLUDED.embedding,
      updated_at = NOW()
    `,
    [chunk.chunkId, chunk.source, chunk.chunkIndex, chunk.title ?? null,
     chunk.content, JSON.stringify(chunk.metadata), DatabaseService.toVector(embedding)],
  );
}

async deleteStale(source: string, keepChunkIds: string[]): Promise<number> {
  const rows = await this.db.query<{ chunk_id: string }>(
    `DELETE FROM chunks WHERE source = $1 AND chunk_id <> ALL($2::text[]) RETURNING chunk_id`,
    [source, keepChunkIds],
  );
  return rows.length;
}

Upsert alone is not enough for safe re-ingestion. Say a document used to have 10 chunks and the new version has 6. Upsert updates the first 6 rows, but the last 4 stay there with their old embeddings and keep being retrieved. The chatbot ends up answering from content that no longer exists.

deleteStale deletes by exclusion after the upsert finishes, instead of "delete everything, then write again". That way search never sees a window where the document is missing.

10.4 IngestService: orchestration only

Abridged version (in the repo, embed + upsert live in a separate embedAndStore method):

async ingestDocuments(docs: RawDocument[]): Promise<IngestReport> {
  const report = { documents: 0, chunks: 0, deleted: 0 };

  for (const doc of docs) {
    const chunks = this.chunker.chunk(doc);
    if (!chunks.length) continue;

    // Embed in batches, then upsert each chunk
    for (let i = 0; i < chunks.length; i += this.config.ingest.embedBatchSize) {
      const batch = chunks.slice(i, i + this.config.ingest.embedBatchSize);
      const embeddings = await this.llm.embedDocuments(batch.map((c) => c.content));
      for (const [j, chunk] of batch.entries()) await this.repo.upsert(chunk, embeddings[j]);
    }

    report.deleted += await this.repo.deleteStale(doc.source, chunks.map((c) => c.chunkId));
    report.documents += 1;
    report.chunks += chunks.length;
  }
  return report;
}

The CLI script scripts/ingest.ts boots Nest as an application context (no HTTP server) and calls the service:

const app = await NestFactory.createApplicationContext(AppModule, { logger: ['log', 'warn', 'error'] });
const report = await app.get(IngestService).ingestDirectory(process.argv[2]);
console.log(`done: ${report.documents} documents, ${report.chunks} chunks, ${report.deleted} stale chunks deleted`);
await app.close();

11. Step 6: Hybrid search (vector + full-text) merged with RRF

11.1 Why you need both branches

Each branch fails exactly where the other is strong:

  • Dense (vector) handles synonyms and paraphrasing well. The user asks "scanner doesn't recognize the code", the doc says "barcode cannot be read", and the vectors still match. But for error codes, product codes, and proper names, vectors are noisy: E-4021 and E-4012 are almost identical in embedding space.
  • Keyword (full-text) is the opposite: it matches E-4021 exactly, but if no token overlaps, the right chunk never shows up.

You combine the two because they cover for each other, not because more branches are always better.

11.2 The dense branch 

async search(queryVector: number[], limit: number): Promise<Candidate[]> {
  const rows = await this.db.query<any>(
    `
    SELECT chunk_id AS "chunkId", source, title, content, metadata,
           1 - (embedding <=> $1::vector) AS score
    FROM chunks
    WHERE embedding IS NOT NULL
    ORDER BY embedding <=> $1::vector
    LIMIT $2::int
    `,
    [DatabaseService.toVector(queryVector), limit],
  );
  return rows.map((r) => ({ ...r, score: Number(r.score) }));
}

<=> is pgvector's cosine distance operator, so 1 - distance is cosine similarity. The query deliberately has just ORDER BY embedding <=> $1 LIMIT n, because that is the shape the planner needs to use the HNSW index. If you add filters, check again with EXPLAIN, since it is very easy to turn this into a full table scan.

11.3 The keyword branch 

async search(queryText: string, limit: number): Promise<Candidate[]> {
  const rows = await this.db.query<any>(
    `
    WITH tsq AS (SELECT websearch_to_tsquery('simple', $1) AS q)
    SELECT c.chunk_id AS "chunkId", c.source, c.title, c.content, c.metadata,
           ts_rank(c.search_vector, tsq.q) AS score
    FROM chunks c, tsq
    WHERE c.search_vector @@ tsq.q
    ORDER BY score DESC
    LIMIT $2::int
    `,
    [queryText, limit],
  );
  return rows.map((r) => ({ ...r, score: Number(r.score) }));
}
  • websearch_to_tsquery accepts the syntax users already know ("exact phrase", a OR b, -exclude) and never throws a syntax error. Passing it as a bind parameter is safe enough; no extra escaping needed.
  • ts_rank is not BM25 (Postgres has no built-in BM25). With RRF that doesn't matter, as you'll see right below.

11.4 Merging two lists: why not add the scores

The "obvious" approach is score = 0.7 × cosine + 0.3 × ts_rank. It's wrong:

  • Cosine lies in [0, 1], while ts_rank is typically around 0.0x. Add them directly and the keyword branch contributes almost nothing. You think hybrid search is running, but it is really just vector search.
  • Switch to real BM25 (scores 0-20, unbounded) and the error flips the other way: the keyword branch dominates.

Reciprocal Rank Fusion (RRF) throws away the raw scores and uses only ranks:

RRF(d) = Σ        1 / (k + rank_branch(d))        k = 60
        branch

A worked example:

Dense:   1. chunk_A   2. chunk_B   3. chunk_C
Keyword: 1. chunk_C   2. chunk_A   3. chunk_D

chunk_A: 1/61 + 1/62 = 0.03252   ← rank 1
chunk_C: 1/63 + 1/61 = 0.03226   ← rank 2
chunk_B: 1/62        = 0.01613
chunk_D: 1/63        = 0.01587

chunk_A wins without being first in either branch, because it ranks well in both. Agreement between two independent methods is a stronger signal than a high score from one.

The constant k flattens the curve: with k = 60, the gap between rank 1 and rank 2 is only about 1.6%, so appearing in several lists matters more than the exact rank.

export function reciprocalRankFusion(
  branches: { dense: Candidate[]; keyword: Candidate[] },
  k = 60,
  topK = 5,
): SearchResult[] {
  const byId = new Map<string, SearchResult>();

  const ingest = (candidates: Candidate[], branch: 'denseRank' | 'keywordRank') => {
    candidates.forEach((candidate, i) => {
      const rank = i + 1;
      const existing = byId.get(candidate.chunkId);

      if (existing) {
        existing.rrfScore += 1 / (k + rank);
        existing[branch] = rank;
        return;
      }

      const { score: _ignored, ...rest } = candidate;
      byId.set(candidate.chunkId, {
        ...rest,
        rrfScore: 1 / (k + rank),
        denseRank: null,
        keywordRank: null,
        [branch]: rank,
      } as SearchResult);
    });
  };

  ingest(branches.dense, 'denseRank');
  ingest(branches.keyword, 'keywordRank');

  return [...byId.values()].sort((a, b) => b.rrfScore - a.rrfScore).slice(0, topK);
}

The result keeps each chunk's denseRank and keywordRank. When debugging, one look tells you whether a result came from semantics, keywords, or both.

11.5 SearchService: two parallel queries, over-fetching 

async hybridSearch(query: string, options: SearchOptions = {}): Promise<SearchResult[]> {
  const topK = options.topK ?? this.config.search.topK;            // 5
  const limit = options.candidateLimit ?? this.config.search.candidateLimit; // 50

  const queryVector = await this.llm.embedQuery(query);

  const [dense, keyword] = await Promise.all([
    this.dense.search(queryVector, limit),
    this.keyword.search(query, limit),
  ]);

  return reciprocalRankFusion({ dense, keyword }, this.config.search.rrfK, topK);
}

Two details decide the quality:

  1. The two branches are two independent queries. A single SQL query that computes both cosine and ts_rank for every row and then sorts still ranks correctly, but no index is used and Postgres has to scan the whole table. Split apart, each branch uses its own index (HNSW / GIN).
  2. Over-fetching. You need 5 final results, yet each branch still fetches 50. The best chunk after fusion is often not first in either branch; it ranks reasonably well in both. Fetch too few and it falls off the lists before it can be merged.

Adding a third branch (multi-query, or real BM25) is just one more array passed to RRF. No score normalization, no weight tuning.

12. Step 7: Designing the prompt so caching works

12.1 System prompt: the static part

The repository ships this prompt in Vietnamese (with <ngu_canh> / <cau_hoi> tags); the English equivalent is shown here.

export const SYSTEM_PROMPT = `You are a Q&A assistant that answers from the provided documents.

Answering rules:
- Answer only from the CONTEXT provided. Do not guess, and do not use knowledge
  outside the context.
- If the context is not enough to answer, say plainly that the information was
  not found. A wrong answer is more dangerous than "I don't know".
- Cite sources with [number] matching the position of the context passage you
  relied on.
- Answer in the same language as the question.
- Be concise and get straight to the point. Do not start with "Based on the documents...".`;

Three rules for this file:

  1. No dynamic values here (dates, user names, session ids). Changing one byte invalidates the cache for everything after it.
  2. No conditional string building (if (x) prompt += ...). Every combination of conditions is a different prefix, i.e. a different cache entry.
  3. Keep it stable across deploys. Editing this file invalidates every existing cache entry.

The rule "answer in the same language as the question" is essential: when documents are in English and the user asks in another language, without this rule the model tends to answer in the language of the context.

12.2 Few-shot: teaching both how to answer and how to decline 

export const EXAMPLES: Example[] = [
  {
    question: 'How long is the product warranty?',
    context:
      '[1] Warranty policy\nAll products are covered by a 12-month warranty from the date of purchase.\n\n' +
      '[2] Returns\nDefective products can be exchanged within the first 7 days.',
    answer: 'Products are covered by a 12-month warranty from the date of purchase [1].',
  },
  {
    // Negative example: there is context, but no passage answers the question.
    question: 'Does the company have a branch in Da Nang?',
    context: '[1] Warranty policy\nAll products are covered by a 12-month warranty from the date of purchase.',
    answer: 'The available documents do not mention any branch information.',
  },
];

Examples are the strongest signal in a prompt: the [1] citation format is easy to misread when described in words, but obvious from a single example.

The refusal example matters as much as the answer example. If every example is a successful answer, the model learns that there must always be an answer. Example #2 teaches it to decline when the context is irrelevant.

12.3 PromptBuilder: ordering by decreasing stability

Prompt caching is a prefix match: the system hashes the prompt from the start up to the cache_control marker. Changing one byte at position N invalidates the cache for everything from N onward. Consequence: content must be ordered from most stable to most volatile.

┌─ system ─────────────────────────────────── CACHED ─┐
│ [0] SYSTEM_PROMPT   persona + rules, never changes  │
│ [1] EXAMPLES        few-shot, never changes         │ ← cache_control
└─────────────────────────────────────────────────────┘
┌─ user ───────────────────────────────── NOT CACHED ─┐
│ [0] CONTEXT         retrieved chunks, per question  │
│ [1] QUESTION        the question, every turn        │
└─────────────────────────────────────────────────────┘

@Injectable()
export class PromptBuilder {
  build(question: string, results: SearchResult[]): GenerateRequest {
    const system: PromptBlock[] = [
      { text: SYSTEM_PROMPT, cacheable: true },
      { text: renderExamples(), cacheable: true },
    ];

    const user: PromptBlock[] = [
      { text: `<context>\n${this.renderContext(results)}\n</context>` },
      { text: `<question>${question}</question>` },
    ];

    return { system, user };
  }

  /** Number from 1 so the model cites [1], [2]... consistent with the few-shot. */
  private renderContext(results: SearchResult[]): string {
    if (!results.length) return '(no relevant documents found)';
    return results
      .map((r, i) => `[${i + 1}] ${r.title ? `${r.title}\n` : ''}${r.content}`)
      .join('\n\n');
  }
}

Why is the retrieved context NOT in the cached region? Each question retrieves a different set of chunks. Put it before the marker and every request creates a new cache entry that no later request can read. You end up paying the cache write premium (about 1.25x the input price) without ever getting the cache read price (about 0.1x or lower, depending on the model).

Cache economics: with a 5-minute TTL, you break even on the second request (1.25 + 0.1 = 1.35 versus 2x without caching). A chatbot with a steady stream of users is the ideal case.

Minimum length: a prefix shorter than the model's minimum cacheable length is not cached, and no error tells you so. The minimum varies by model (from a few hundred to a few thousand tokens). The system prompt + few-shot above is fairly short, so check cacheWriteTokens on the first request. If it is 0, the prefix is too short: add business rules, add few-shot examples, or accept running without cache.

Why <context> and <question> tags: XML tags mark a clear boundary between data and question, which helps the model avoid treating document content as instructions.

13. Step 8: Chat service and HTTP API

13.1 ChatService: one RAG loop 

@Injectable()
export class ChatService {
  constructor(
    @Inject(LLM) private readonly llm: LlmService,
    private readonly search: SearchService,
    private readonly prompt: PromptBuilder,
  ) {}

  async ask(question: string): Promise<ChatResponse> {
    const results = await this.search.hybridSearch(question);    // 1. retrieve
    const request = this.prompt.build(question, results);        // 2. build prompt
    const { text, usage } = await this.llm.generate(request);    // 3. generate

    return {
      answer: text,
      citations: results.map((r, i) => ({
        index: i + 1,            // matches [n] in the answer
        chunkId: r.chunkId,
        source: r.source,
        title: r.title,
      })),
      usage,
    };
  }
}

13.2 Controller 

class AskDto {
  @IsString()
  @IsNotEmpty()
  @MaxLength(2000)
  question!: string;
}

@Controller('api')
export class ChatController {
  constructor(
    private readonly chat: ChatService,
    private readonly search: SearchService,
    private readonly repo: ChunkRepository,
    private readonly db: DatabaseService,
  ) {}

  @Get('health')
  async health() {
    return { status: (await this.db.ping()) ? 'ok' : 'degraded' };
  }

  @Post('chat')
  ask(@Body() dto: AskDto) {
    return this.chat.ask(dto.question);
  }

  /** Inspect retrieval results directly, without the LLM. Useful when tuning. */
  @Get('search')
  search_(@Query('q') q: string, @Query('mode') mode?: string) {
    if (mode === 'dense') return this.search.denseSearch(q);
    if (mode === 'keyword') return this.search.keywordSearch(q);
    return this.search.hybridSearch(q);
  }

  @Get('stats')
  stats() {
    return this.repo.stats();
  }
}

Besides validation, @MaxLength(2000) stops users from sending huge questions that inflate token costs.

13.3 main.ts 

import 'dotenv/config';

async function bootstrap(): Promise<void> {
  const app = await NestFactory.create(AppModule);
  const config = app.get<AppConfig>(CONFIG);

  app.useGlobalPipes(
    new ValidationPipe({ whitelist: true, forbidNonWhitelisted: true, transform: true }),
  );
  // Let the pool close cleanly on SIGTERM instead of cutting off running queries.
  app.enableShutdownHooks();

  await app.listen(config.port);
}
bootstrap();

14. Step 9: End-to-end test run 

cp .env.example .env      # fill in AWS credentials (or use AWS_PROFILE)
docker compose up -d      # Postgres + pgvector
npm install
npm run db:init           # create extension, table, indexes

Put a few markdown files in data/in/, for example data/in/warranty.md:

# Warranty Policy

## Warranty period
All products are covered by a warranty of 12 months from the date of purchase.

## Returns
Products with a manufacturing defect can be exchanged within the first 7 days.

npm run ingest
npm run start:dev

Check retrieval before spending generation tokens. Make this a habit from day one when tuning RAG:

npm run search -- "warranty 12 months"            # hybrid, with per-branch ranks
npm run search -- "warranty 12 months" dense      # vector only
npm run search -- "warranty 12 months" keyword    # full-text only

Hybrid output prints rrf=... dense=1 keyword=2 <source> for each chunk, so you can see at a glance which branch a chunk came from.

Ask the chatbot:

curl -s -X POST http://localhost:8000/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"question": "How long is the product warranty?"}'

The response looks like this (illustrative values):

{
  "answer": "All products are covered by a 12-month warranty from the date of purchase [1].",
  "citations": [
    { "index": 1, "chunkId": "data/in/warranty.md#0", "source": "data/in/warranty.md", "title": "Warranty period" },
    { "index": 2, "chunkId": "data/in/warranty.md#1", "source": "data/in/warranty.md", "title": "Returns" }
  ],
  "usage": {
    "inputTokens": 180,
    "outputTokens": 25,
    "cacheWriteTokens": 0,
    "cacheReadTokens": 650
  }
}

Send the same question twice: the first request has cacheWriteTokens > 0, the second has cacheReadTokens > 0. If cacheReadTokens stays 0 across many requests, something is breaking the cache. It's usually a dynamic value that slipped into the system prompt, or a prefix below the minimum length.

Other endpoints:

  • GET /api/search?q=...&mode=hybrid|dense|keyword: inspect retrieval results.
  • GET /api/stats: chunk count per source.
  • GET /api/health: health check (used by Docker).

15. Step 10: Tests, production Docker, and CI

15.1 Unit tests for the pure logic

The chunker and RRF are pure functions, so their tests need no database and no AWS:

it('a chunk ranked well in both branches beats a chunk ranked first in only one', () => {
  const results = reciprocalRankFusion(
    {
      dense: [candidate('dense-only'), candidate('both')],
      keyword: [candidate('keyword-only'), candidate('both')],
    },
    60,
    3,
  );
  expect(results[0].chunkId).toBe('both');
});

it('splits by heading and prepends the heading to chunk content', () => {
  const chunks = makeChunker(1000).chunk({
    source: 'doc.md',
    text: '# Title\n\nIntro.\n\n## Part A\n\nContent A.\n\n## Part B\n\nContent B.',
  });
  expect(chunks).toHaveLength(3);
  expect(chunks[1].title).toBe('Part A');
  expect(chunks[1].content).toContain('Part A');
});

15.2 Multi-stage Dockerfile 

FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY tsconfig.json tsconfig.build.json nest-cli.json ./
COPY src ./src
RUN npm run build
RUN npm prune --omit=dev

FROM node:20-alpine AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder --chown=node:node /app/node_modules ./node_modules
COPY --from=builder --chown=node:node /app/dist ./dist
COPY --chown=node:node package.json ./
USER node
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
  CMD node -e "require('http').get('http://127.0.0.1:'+(process.env.PORT||8000)+'/api/health',r=>process.exit(r.statusCode===200?0:1)).on('error',()=>process.exit(1))"
CMD ["node", "dist/main"]
  • The npm ci layer is separate so the Docker cache survives source edits.
  • The runtime runs as the node user, not root.
  • The healthcheck uses Node's built-in http, so there's no need to install curl.

15.3 Production Compose: fail fast on missing variables

docker-compose.prod.yml layers on top of the dev file, adds the app service, hides the Postgres port, and makes every sensitive value mandatory:

services:
  postgres:
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required}
    ports: !reset []   # Postgres not exposed outside the network

  app:
    build: .
    environment:
      DB_HOST: postgres
      DB_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required}
      BEDROCK_GENERATION_MODEL_ID: ${BEDROCK_GENERATION_MODEL_ID}
      # ...
    depends_on:
      postgres:
        condition: service_healthy

docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

The ${VAR:?message} syntax makes the deploy fail immediately when a variable is missing, instead of silently running with the default postgres password.

15.4 CI with GitHub Actions 

name: CI
on:
  push:
    branches: [main]
  pull_request:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm test
      - run: npm run build

16. Production Pitfalls That Quietly Destroy RAG Quality

None of these throw an exception. The chatbot just answers worse, and the cause is very hard to track down.

  1. Wrong input_type when embedding. Using search_document for questions → recall drops a few percent, and no log tells you.
  2. Mismatched vector dimensions. Switching embedding models without updating EMBEDDING_DIMENSIONS and vector(N) → inserts fail (the lucky case, at least there's an error). Switching to a model with the same dimensions without re-embedding the corpus → old and new vectors share one index, results are meaningless, and there is no error at all.
  3. Unicode NFC/NFD in accented languages such as Vietnamese. The character ề has two valid encodings that look identical on screen but are not equal to a computer. Documents copied from PDFs or created on macOS are often NFD → full-text search doesn't match. Fix: text.normalize('NFC') at both ingest time and query time, inside one shared function.
  4. Full-text tokenizers for Vietnamese/Japanese. Postgres splits on whitespace, so the Vietnamese word "kiểm kê" (inventory check) becomes two tokens kiểm + kê, and also matches documents containing "kê khai" (declaration). Japanese has no spaces at all → the keyword branch is nearly useless; you need word segmentation in the application layer (MeCab, Sudachi) or an extension such as PGroonga.
  5. Adding un-normalized scores from the two branches. Hybrid "runs" but only one branch actually matters. The test: if hybrid isn't better than the best single branch on its own, the problem is the fusion step; don't go tuning weights.
  6. Prompt cache not hitting. A dynamic value slipped into the system prompt, or the prefix is below the minimum length. Always watch cacheReadTokens.
  7. Re-ingest without cleaning old chunks. The chatbot answers from content that has been removed from the documents.

17. From here to production: the upgrade path

By this point, the system is already capable of answering questions over a document collection.

The next challenge is not functionality.

The next challenge is reliability.

Most production RAG systems evolve through a series of small improvements rather than a single architectural rewrite.

The following upgrades are listed roughly in order of value delivered relative to implementation effort.

The codebase above is Advanced RAG at its minimum: hybrid search + RRF + header injection + prompt caching. When you put it in front of real customers, these are the next steps, ordered by benefit per unit of effort:

  1. A golden set before any optimization. 30-50 (question, correct chunk) pairs taken from real logs, not questions you made up (made-up questions are always easier than reality). Measure Recall@5 and Recall@50. Without measurement, every "improvement" is a hunch.
  2. Rewrite questions using conversation history. On turn 2 the user asks "what if it can't read it?". Searching with that sentence as-is finds almost nothing. One LLM call to rewrite it into a standalone search query is the biggest improvement for multi-turn chatbots.
  3. Confidence threshold + escalation path. Retrieval always returns k results even when the corpus has no answer, and that is the most common source of hallucination. Note that the RRF score cannot be used as a threshold (it only reflects rank); keep the raw cosine separately to measure confidence.
  4. Verify citations in code. Ask the model to return JSON with citations as a list of chunk_id, then check every id is in the set you just sent. A few lines of code that catch most blatant fabrications.
  5. Output guardrails. A layer that runs after the LLM answers and before the customer sees it: check for PII, unknown URLs, out-of-scope topics, length. Many systems go to production without this layer.
  6. Handle stop_reason. Before reading content, check stop_reason (max_tokens = the answer was truncated; some newer models may return refusal). Return an appropriate message instead of an empty or half-finished string.
  7. Stream answers to the client (InvokeModelWithResponseStreamCommand) to reduce perceived waiting time.
  8. Parent-child chunking for summary questions like "what are the steps of this process".
  9. Reranker: only once Recall@50 is good but the top-5 ordering is still messy. This step adds the most latency.
  10. Observability: log latency per stage (embed / search / generate), tokens, and cache hit rate per request.
  11. Loosen the keyword branch query. websearch_to_tsquery joins every word with AND, so long questions match almost nothing (see section 3.2). Extract a few key terms (codes, names, nouns) and join them with OR before searching, and the keyword branch will contribute much more.

For multi-tenant setups (several businesses on shared infrastructure), filtering by tenant_id from the schema up is a matter of right/wrong, not quality. Without it, data leaks between customers.

18. Wrapping Up

After building several generations of RAG systems, one lesson keeps repeating itself:

RAG is fundamentally a retrieval problem.

Most production issues are not caused by the language model.

They come from:

  • poor chunking
  • weak retrieval
  • stale documents
  • missing observability
  • broken citations
  • unreliable synchronization

The architecture in this article intentionally avoids unnecessary complexity.

There is no external vector database.

No orchestration framework.

No hidden abstraction layers.

Just PostgreSQL, pgvector, Amazon Bedrock, Hybrid Search, and a clear separation between indexing, retrieval, and generation.

For teams involved in application software development, that simplicity is often an advantage rather than a limitation. A system that can be understood, measured, and debugged is significantly easier to improve than one hidden behind multiple abstraction layers.

A system that can be understood, measured, and debugged is much easier to improve than one that depends on layers of abstraction.

If you're building a RAG chatbot today, invest in retrieval first.

Everything else becomes easier afterwards.

來源：DEV Community · dev.to

#教學/實踐#RAG#部署/工程#Anthropic