理解 RAG：為何 AI 回答前需要先查閲資料

Exploring Retrieval-Augmented Generation (RAG), how it works, and why giving AI access to the right information can be more useful than relying solely on what it already knows.

Generative AI · Applied AI · Software Engineering

Have you ever asked ChatGPT a question and received an answer that sounded incredibly convincing, only to discover later that it was completely wrong?

What's interesting is that AI can explain something with remarkable confidence, even when the information isn't accurate.

At first, I thought the solution was simple: just use a smarter or newer AI model.

But as I started learning more about Generative AI, I realized the problem wasn't that straightforward.

There's a difference between an AI that's good at generating answers and an AI that actually has the right information to answer a question.

That's when I came across an approach called Retrieval-Augmented Generation, or simply RAG.

AI Is Smart, but It Doesn't Know Everything

Imagine having a really smart employee.

They understand many subjects, can explain complicated concepts, and are great at solving problems.

But there's one limitation: they don't know everything about your company.

For example, let's say you ask:

"How many days of annual leave are employees entitled to under our company's latest policy?"

That employee might understand general labor regulations. However, they wouldn't necessarily know your company's internal policies, especially if those policies were updated last week.

The same thing happens with Large Language Models (LLMs).

Models like GPT learn patterns, concepts, and information during training. However, that knowledge doesn't automatically include private company documents, internal policies, or information that became available after their training data was collected.

When a model doesn't have enough information, it may sometimes generate an incorrect answer that still sounds perfectly reasonable.

This phenomenon is commonly known as hallucination.

So, what if we could give AI the company's policy documents before asking it to answer?

That's essentially where RAG comes in.

So, What Exactly Is RAG?

RAG is an approach that allows an AI-powered application to retrieve relevant information from external sources before using that information as additional context to generate a response.

The concept was introduced in a 2020 research paper by Patrick Lewis and his colleagues.

The name might sound complicated, but we can break it down into three simple parts:

• Retrieval: Find information relevant to the question.
• Augmentation: Add that information to the context provided to the AI.
• Generation: Generate an answer using the question and the additional context.

Think of it like taking an exam.

There are two kinds of exams: closed-book and open-book.

In a closed-book exam, you rely entirely on what you've studied and remembered.

In an open-book exam, you're allowed to look things up before answering.

RAG works on a similar principle.

The difference is that the AI doesn't magically know where every piece of information is stored. The application we build is responsible for finding and providing the relevant information.

And here's something important:

RAG doesn't mean retraining the AI model.

We're not permanently adding new knowledge to the model. We're simply giving it additional reading material when it needs it.

How Does RAG Actually Work?

Let's use a simple example.

Imagine we're building an internal chatbot that helps employees find information from company documents.

The company has several documents covering things like annual leave, reimbursement policies, and work-from-home guidelines.

One day, an employee asks:

"How many days of annual leave do I get?"

A simplified RAG process would look something like this:

1. The employee asks a question.

The application receives the question about annual leave.

2. The system searches for relevant information.

Instead of immediately asking the LLM to generate an answer, the application searches the company's knowledge base.

3. The system retrieves relevant content.

It finds a section of the company's leave policy that says:

"Permanent employees are entitled to 12 days of annual leave per year."

4. The information is provided to the LLM.

The application sends both the employee's original question and the retrieved policy information to the model.

5. The LLM generates an answer.

Using the supplied context, the model might respond:

"According to the company's leave policy, permanent employees are entitled to 12 days of annual leave per year."

The policy and numbers above are fictional examples.

Pretty straightforward, right?

But there's an interesting question.

How does the application know which part of the company's documents is relevant to the employee's question?

That's where things get a little more technical.

How Does RAG Find the Right Information?

Let's say the company has 500 PDF documents.

Obviously, sending all 500 documents to the AI every time someone asks a question wouldn't be a great idea.

It would be inefficient, expensive, and potentially exceed the amount of information the model can process in a single request, commonly known as its context window.

Instead, a typical RAG system prepares documents so they can be searched efficiently.

There are a few important concepts behind this.

1. Chunking: Breaking Documents into Smaller Pieces

Imagine having a 300-page book.

If someone asks about annual leave, you wouldn't hand them the entire book and expect them to read everything.

You'd find the relevant page or paragraph.

That's essentially the idea behind chunking.

Large documents are divided into smaller pieces of information called chunks.

However, smaller doesn't always mean better.

If we split a paragraph in the wrong place, we might lose important context.

On the other hand, if a chunk is too large, it might contain lots of unnecessary information.

So, how we divide documents actually matters.

2. Embedding: Turning Text into Numerical Representations

Once documents are divided into chunks, we can use an embedding model to convert each piece of text into a numerical representation.

Why numbers?

Because computers can compare these numerical representations to estimate how closely related two pieces of text are in meaning.

For example, consider these two sentences:

"How many vacation days do employees get?"

"Annual leave entitlement for staff members."

They use different words, but they're talking about nearly the same thing.

Embeddings can help a system recognize this relationship without relying entirely on exact keyword matches.

3. Vector Search: Finding Relevant Information

Those numerical representations can be stored in a database or search index that supports vector similarity search.

When a user asks a question, the question is also converted into an embedding.

The system compares that embedding with the stored document embeddings to find the most relevant chunks.

Then, the original text from those chunks is passed to the LLM as context.

One thing worth mentioning: RAG doesn't always require a vector database.

Traditional keyword search or a combination of keyword and vector search, often called hybrid search, can also be used.

The goal is the same: retrieve information that's actually useful for answering the question.

Does RAG Completely Eliminate AI Hallucinations?

Unfortunately, no.

This was one of the most interesting things I learned.

Initially, I assumed that giving an AI the correct documents would automatically produce correct answers.

But that's not always how it works.

Imagine asking someone to answer a question using a reference book.

Several things could go wrong.

They might open the wrong page.

They might find the correct page but misunderstand the explanation.

Or the book itself might contain outdated information.

The same problems can happen in a RAG system.

For example, imagine an employee asks about the company's leave policy for 2026, but the retrieval system returns a document from 2024.

The AI might generate an answer that accurately reflects the retrieved document.

However, that answer could still be wrong because the information is outdated.

This means the quality of a RAG system doesn't depend entirely on how powerful the LLM is.

It also depends on the quality of the data, how documents are prepared, how relevant information is retrieved, and how the model is instructed to use that information.

RAG can help reduce hallucinations, but it doesn't eliminate them.

And that's an important distinction.

When Should We Use RAG?

As a web developer, I find RAG particularly interesting because it allows us to connect the capabilities of LLMs with data that already exists inside an application.

For example, a company with hundreds of internal documents might want to build a chatbot so employees don't have to search through them manually.

A software engineering team might want an assistant that can answer questions about API documentation, system architecture, or deployment guidelines.

A customer support application could retrieve product documentation before answering customer questions.

These are situations where having access to external information can make AI significantly more useful.

But here's the thing.

Not every AI application needs RAG.

If we're building something that improves grammar, rewrites sentences, or generates creative ideas, retrieving documents might not provide much value.

RAG also introduces additional complexity.

We need to prepare and maintain our data, manage a search index, handle document updates, and retrieve information when it's needed.

All of these can affect application complexity, cost, and response time.

So instead of immediately asking, "Can we use RAG for this project?"

I think a better question is:

"Does our application actually need to retrieve external information dynamically to produce useful answers?"

If the answer is yes, RAG might be a good fit.

If not, adding RAG could simply make the system more complicated than necessary.

RAG Isn't the Only Approach

After learning how RAG works, I started thinking about a few other questions.

What if the documents aren't that large?

What if users repeatedly ask questions about the same information?

Do we really need to search for relevant documents every single time someone asks a question?

These questions led me to another approach called Cache-Augmented Generation (CAG).

While RAG focuses on retrieving relevant information when it's needed, CAG explores the idea of preparing context ahead of time and reusing it through caching mechanisms.

Both approaches have their own benefits, limitations, and trade-offs.

And I think comparing them is interesting because it's not simply about choosing which technology is better.

It's about understanding how different approaches fit different data characteristics and application requirements.

That's what I plan to explore in the next article.

Final Thoughts

Learning about RAG changed the way I think about building AI-powered applications.

Previously, I tended to assume that the quality of an AI application mostly depended on the model behind it.

A smarter model should produce better results, right?

Well, not necessarily.

RAG shows that the quality of an AI system also depends on how we provide information to that model.

Even a powerful model can produce a poor answer when the context is wrong.

On the other hand, giving a model access to relevant, well-organized information can significantly improve its chances of generating a useful response.

As a software developer, this idea feels surprisingly familiar.

We're already used to building systems that retrieve data, process it, and return useful results to users.

RAG brings a similar pattern into Generative AI, with the LLM acting as a component that interprets context and generates natural-language responses.

Ultimately, building an AI application isn't just about making a model capable of answering questions. It's also about making sure the model has the right information to answer them.

And for me, that's one of the most important lessons from understanding RAG.

References & Further Reading

• Lewis et al. (2020) — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Microsoft Learn — Retrieval-Augmented Generation and Indexes
• Microsoft Architecture Center — Design and Develop a RAG Solution

This is the first article in my RAG vs. CAG exploration series, written from the perspective of a software developer learning more about Generative and Applied AI.