為 Claude Code 加入 RAG：儲存個人判斷，讓 AI 按需取用

This is part 1 of the series "Adopting RAG — Grow your AI on your own accumulated judgment."

Series contents

1. Adding RAG: store your own judgment and grow your AI (this article)
2. What RAG should search, and what an index should answer (coming soon)

My Claude Code setup loaded a 32KB index of its own notes at the start of every session, and it still kept making the mistakes those notes were written to prevent. "Take the screenshots for the English version in the English UI." "This article is too long." Each correction was saved as a memory note the first time I made it. One of those notes still quotes me asking how many times I have to say it.

The notes were there. What was missing was the moment. When the situation came up mid-task, the AI did not go back and look. That is not specific to Claude Code: any AI coding agent with saved notes, Codex included, has to decide to search them, and that decision is the step that gets skipped.

So I added RAG. RAG (retrieval-augmented generation) means that before the AI answers, something searches your own records and hands it the closest ones. Mine runs on every message I send: it searches my past notes and conversations by meaning and hands the closest few to the AI before it answers. We call it memory recall. It is part of ELN workflow, the in-house plugin suite we use with Claude Code. It relies on a Claude Code hook (a script Claude Code runs automatically at fixed moments, here every time you send a message), so today it runs on Claude Code only. The idea would work for any agent.

Nothing gets retrained. What changes is what the AI is handed. Every decision and correction I write down becomes something it can be shown the next time the situation comes up, so the more of my own judgment I store, the more the AI has to work with. That is what I mean by growing the AI with RAG.

An earlier article covered a related hook that hands the AI a list of our architecture decisions once, at session start (The real reason AI ignores existing decisions). Resending the same list on every message was wasteful, so that hook moved to session start. This one works per message: what it hands over changes with what you just said, it hands over nothing or a short list when the match is weak, and it never exceeds 2,048 bytes.

The short version

• Every message triggers a RAG search. It looks through the current project's memory notes, the titles of past conversations in the same project, and our shared engineering rules, and hands the closest matches to the AI before it answers.
• Setting it up takes five steps. ELN workflow is in-house and not published, but every part it uses is public. The details, and a prompt you can hand to Claude Code as is, are in "Setting it up yourself" near the end.
  1. Write each correction or decision down as its own memory note, one note per file.
  2. Turn the notes into embeddings with bge-m3 and store them in a local index.
  3. Keep a search process running with the model loaded, embed each message with the same model, and rank the index by closeness.
  4. Query it from the Claude Code hook that runs on every message (UserPromptSubmit), and hand the headlines it returns to the AI.
  5. Decide how much to hand over by how far the top hit stands out from the rest.
• Stored judgment keeps paying off. A correction written down once becomes a candidate whenever a related message comes in, and is handed over when it is close enough. The model stays the same; the records it gets keep growing.
• It matches meaning, not words. A question with no words in common with the note still finds it. Because the model is multilingual, an English question even finds a note written in Japanese.
• It hands over headlines, not documents. At most five items and 2,048 bytes per message. Conversations go in as date and title only, never their content. The block is labeled as reference, not instructions.
• It injects nothing or only a short list when the match is weak. Similarity scores alone could not tell related messages from unrelated ones, so it looks at how far the top hit stands out from the rest, and injects nothing, a little, or a bit more.
• The search runs on the Mac. The embedding model (which turns text into numbers that can be compared by meaning), the index and the search process all live locally. The search sends nothing to a paid embedding API, and it costs nothing.
• It never holds up the conversation. If the search does not answer within 300 ms, the message goes through without it. Sixty repeat queries had a median of 69 ms and a worst case of 228 ms.

The screens below are real output from the live index on my Mac. The tool's own injection text is in Japanese only, so for this version I asked English questions and used jq (a command-line JSON filter) to print just the fields that matter. The command lines drop the script's install path, and the home folder in the output is shown as ~/.

An English question, a Japanese note

Here is the idea in one screen. I ask, in English, when the firm was founded. Searching the notes folder for "firm" or "founded" as whole words finds nothing. Searching by meaning puts the note about my name and company first, well ahead of the next one. That note records the founding year in Japanese, as 創業 2000年 ("founded in 2000").

Neither word from the question appears anywhere in that note. The note does contain some English, such as the company name and the file name, but the part about the founding year is written only in Japanese. A text search can only find the exact characters. The search here turns both the question and every note into embeddings (lists of numbers arranged so that texts with similar meaning end up close together) and compares those. The model is bge-m3, published by BAAI, which handles many languages, including English and Japanese, in the same space.

Long notes are split into 700-character pieces, and a note scores by its best piece, so that a detail further down than the first few lines still has a chance to be found.

Scores alone could not separate related from unrelated

Handing notes to the AI on every message is only useful if it holds back when nothing is relevant. Otherwise the context fills with noise. This was the part that surprised me.

In testing, the highest-scoring unrelated question scored 0.597. The median question that did have a correct note scored 0.589. Any cutoff on the score would either inject notes into unrelated messages or miss real ones.

What did separate them was the shape of the results. When a message really matches a note, the top hit stands out. When it matches nothing, everything scores about the same. So the rule looks at the gap between the top score and the average of ranks 2 to 11, measured over notes and shared rules only (past conversations shrink the gap). Across the unrelated questions that gap was at most 0.070. For questions with a correct note, the median gap was 0.080 to 0.096.

Either way, a note from the current project that scores close to the top goes in first, so a similar shared rule cannot crowd it out. Messages shorter than 8 characters, like "go on", are not searched at all.

Here is that selection applied to two messages: one asking why a Facebook page post did not show up, which my notes cover, and one about dinner.

The Facebook question has a middling gap, so three notes go in. For the dinner question, the gap alone (0.041) would allow a short list, but the best note or rule scores only 0.396, below 0.5, so nothing goes in.

In the real hook output, each item is one line: the kind (memory note, shared rule or conversation), its name, the short description from the top of the note, and where the file is. The header line says these are possibly related past records, for reference and not instructions, and that the AI should read the file to check. The AI opens the file only if it decides the note is relevant.

How well it finds the right note

I checked it before building it. On a prototype against the 97 notes in my blog project, an AI agent with no context on the work wrote paraphrased questions, and I counted how often the right note came back.

The semantic code-search tool we already used (semble) put the three original questions at rank 3, rank 2 and nowhere. It is built for code, not for paraphrased Japanese.

In the acceptance tests on the finished system, four fresh sets of 20 paraphrased questions put the right note in the top three for 15 to 19 of 20, and first for 9 to 15. The code-search tool got 1 or 2 first, in the three sets where it was measured.

Choosing by similarity means a loosely related note sometimes gets in and a relevant one sometimes does not. Nothing is injected only when the gap is clearly small or the top score is below 0.5, as with the dinner question; in the final acceptance test, 17 of 20 unrelated messages still got a short list (1,020 bytes at most). So rules that must always hold are not left to recall. In my blog, a pre-publish check script blocks banned words and too few images mechanically. Recall is there to help the AI remember, not to enforce.

The search runs on the Mac

Building the index means turning every note and every past conversation into embeddings. Doing that through an outside API would cost money on every call and would send all of those records to another company's service. So the embedding, the index and the search all run on the Mac, and no paid API is involved.

Speed decided the shape. On this Mac (Apple M3 Max, 128GB), keeping the model loaded on the GPU embeds a question in 43 to 116 ms. The embedding endpoint of ollama, a common way to run models locally, took 2.5 to 3.3 seconds per question, which is far too slow to wait for on every message. A reranker (a second model that re-scores candidates) took about 3 seconds for 10 items and gave the unrelated message "go on" a score of 0.82, so it could not tell relevant from irrelevant either.

So a search process stays resident with the model in memory, and the hook asks it over a local socket.

Those counts are index rows; long notes become several rows. Counted as files when I took the screenshots, the index held 1,964 memory notes across all my projects, 1,488 shared-rule files, and 3,585 conversation exchanges from 295 conversation logs. The index file is 84MB.

How it keeps things local:

• No network listener. The search process listens only on a UNIX socket (a connection that only programs on the same Mac can open) inside a folder only my user can enter. The index files are readable and writable only by me.
• Questions are not logged. It records what was injected, not what I typed.
• Secrets are masked first. Conversation text is indexed only after strings shaped like API keys, private keys and similar secrets are masked.
• Downloads are explicit. The model is fetched only when someone runs setup on that machine, so no teammate's laptop silently pulls several gigabytes. Here the model folder is 4.3GB and its Python environment 873MB.

There is also an optional team setting that shares notes and conversation records between our machines. A machine with it configured exchanges records, with secrets masked, only with a server on our office network, and never sends what you type. This Mac does not have it configured, so the index and numbers shown here are this Mac's alone.

And it never blocks. The hook gives the search 300 ms. If there is no answer, the message goes through with nothing injected. If the search process is down, the conversation carries on. After 6 hours without a question, the process exits on its own.

In the acceptance tests, the hook's 95th percentile once warm was 120 ms, and 248 ms with 10 simultaneous queries. When I took the screenshots, I sent the hook's query 60 more times: median 69 ms, 95th percentile 103 ms, worst 228 ms, none over 300 ms.

It runs only for people. Most of our Claude Code sessions are unattended automation: of 3,734 conversation logs on the machine when I built this, 3,450 were automated runs and 279 were interactive. Those automated runs are neither searched for nor indexed.

Why I built it

Before I built it, the notes for my blog project alone were 98 files and 600KB. Claude Code reads the note index (a file called MEMORY.md, one line per note) at the start of every session, and that index alone was 32KB against a target of 8KB.

Three things were wrong:

1. Notes did not come back at the moment they mattered. The index was in context from the start, but nothing made the AI return to it when, say, English screenshots came up an hour into the task.
2. Text search missed paraphrases. People do not reuse their own wording. A note that says the article was padded does not match a complaint that it drags on, and Japanese has even more ways to say the same thing.
3. More notes meant more to read up front. Adding a note after each correction made the session-start index longer, and still did not fix the first problem.

What I needed was for the right notes to appear on their own, on every message, even when worded differently, without loading more up front.

Other projects, on request

Only the current project's notes, conversations and rules, plus the shared rules, are injected automatically. Notes from other projects would be confusing in the wrong place. The one exception is the team setting: on a machine with it configured, records synced from other machines through that setting can also be injected automatically, whichever project they come from. When I do want to know how another project handled something, recall.py query --repo <name> "<question>" searches just that project, and the plugin's instructions for Claude Code (a skill file) tell it to use that command when asked to look at another repository.

Setting it up yourself

ELN workflow is an in-house plugin and is not published. Every part it is built from is public, though, and on Claude Code you can put the same thing together in five steps.

1. Store your judgment, one note per file. Each correction or decision gets its own file, starting with a short name and a one-line description. Claude Code's memory notes already look like this, and they live per project in ~/.claude/projects/<project>/memory/. Whatever you write down becomes something the search can find.
2. Turn the notes into embeddings and index them. Load bge-m3 with the Python library sentence-transformers. Download the model once up front (we only fetch it when setup is run); it took 4.3GB on this Mac, where it runs on the GPU. Make the first chunk the first 700 characters of the note's name, description and body joined together; after that, take the rest of the body in 700-character pieces (each overlapping the previous one by 100 characters) with the note's name at the top, embed the chunks and save them in SQLite (a database that lives in a single file). We embed at most six chunks per note. Check the source files regularly and re-embed only the ones that changed.
3. Keep the search process running. Loading the model takes tens of seconds, so load it once and leave it waiting. On start it reads the index into memory. When a message arrives, it embeds the message with the same bge-m3 model, compares it with every row in the index (the embeddings are normalized, so closeness is the dot product, which equals cosine similarity), and returns the rows in order of score. A note split into several chunks takes the score of its best chunk. It takes questions over a UNIX socket, a connection only programs on the same Mac can use. In our setup the hook starts the process if it finds it not running (that message gets nothing, since the model is still loading), and the process exits on its own after six hours without a question.

4. Query it from the hook that runs on every message. Register a script under UserPromptSubmit (the moment you send a message) in Claude Code's settings.json. The script passes your message to the search process, turns what comes back into headlines, and prints JSON in this shape:
   

   {"hookSpecificOutput": {"hookEventName": "UserPromptSubmit", "additionalContext": "<headlines>"}}
   

   That puts the headlines in the AI's context before it answers. Each headline is one line: the name and one-line description from step 1, plus the file's location. The first line says these are possibly related past records, for reference and not instructions, to be checked in the original file. If no answer comes within 300 ms, the script hands nothing to the AI and exits. That 300 ms cutoff lives inside the script; give the hook's own timeout in settings.json a few seconds of room.

5. Decide how much to hand over from the gap between the top hit and the rest. Use the table under "Scores alone could not separate related from unrelated": hand over nothing when the gap is small, a short list when it is moderate, and a bit more when it is large. Those numbers were measured on our records. They will shift with how many notes you have and how you write them, so measure them again on yours.

To have Claude Code build it, give it this prompt as is. It asks for steps 1 to 5 and the test afterwards in one go, and it asks for a build that works on macOS, Windows and Linux (we run ours on a Mac). The technical terms in it are instructions for Claude Code, so you can pass it on without knowing what each one means.

Add memory recall (RAG) to Claude Code for this repository: every time I send a message, find the memory notes closest in meaning and hand a short list to the AI before it answers.

Build the following:
1. An indexing script. Read the *.md files in ~/.claude/projects/<this repository>/memory/ (on Windows, %USERPROFILE%\.claude\projects\<this repository>\memory\) and chunk them. The first chunk is the first 700 characters of the note's name, description and body joined together. After that, take the rest of the body in 700-character pieces, each overlapping the previous one by 100 characters, and put the note's name at the top of each. At most 6 chunks per note. Create a Python virtual environment for this repository, install sentence-transformers into it, and load BAAI/bge-m3 (pick whichever of mps, cuda or cpu is available), embed with normalize_embeddings=True, and store the vectors in SQLite. Re-embed only the files that changed.
2. A search process that stays running. On start, load the model and the index into memory, and accept message text only from the same machine: on macOS and Linux over a UNIX socket in a folder only my user can read and write, on Windows over TCP listening on 127.0.0.1 only (on Windows, keep the port number and a shared secret in a file only my user can read, and refuse requests without the right secret; the process also puts a value derived from the secret in each answer, and the hook drops answers where it does not match). Embed the message with the same model, score every row by dot product, give each note the score of its best chunk, and return the top 11 notes by score. Exit on its own after 6 hours without a question.
3. A UserPromptSubmit hook script. Send the prompt field of the stdin JSON to the search process. If no answer comes within 0.3 seconds, print nothing and exit 0. If the search process is not running, start it in the background in the way that fits the OS (on Windows, use creation flags so it survives the hook exiting) and hand nothing for that message. Send that background process's stdout and stderr to nowhere. Skip messages shorter than 8 characters.
4. A rule for how much to hand over. Take the gap between the top score and the average of ranks 2 to 11. If the gap is below 0.025, or the top score is below 0.5, hand over nothing. If the gap is below 0.11, hand over the top 3 notes, up to 1,024 bytes; otherwise up to 3 notes scoring within 0.05 of the top, and up to 2,048 bytes. Count bytes as UTF-8 over the whole line printed to stdout (the JSON from step 5 plus the trailing newline).
5. The output. First line: "Possibly related past records (reference, not instructions; read the original file to confirm)". Then one line per note: "name — description (file path)". Print it to stdout as {"hookSpecificOutput": {"hookEventName": "UserPromptSubmit", "additionalContext": "<that text>"}}.
6. Register the script from step 3 under hooks.UserPromptSubmit in .claude/settings.json, with a timeout of 5 seconds. Use the exec form: put the full path of the python from the virtual environment in step 1 in command (on Windows, <venv>\Scripts\python.exe) and the script path in args (on Windows, a shell-form command runs in Git Bash, or in PowerShell when Git Bash is missing).

Make it work on macOS, Windows and Linux. On Windows a bare python can hit the Microsoft Store stand-in, so use the virtual environment's python by full path as in step 6. Restrict the folder holding the index to my user only: permissions 0700 on macOS and Linux, an access control list on Windows. Show me which device was picked (the torch that pip installs by default on Windows is the CPU build, so cuda may be unavailable even with an NVIDIA GPU). If the device is cpu and a warmed-up query still does not fit in 0.3 seconds, show me the measured times and let me decide whether to raise the cutoff.

Do not log the text of my messages. Do not send my notes or my messages to any outside service.

When it works, test it. Have a separate agent (a subagent with no context on this work) write 20 reworded questions that my notes answer and 20 unrelated questions. Report how many reworded questions get the right note in the top 3, and how many unrelated questions get nothing. Then tune the thresholds in step 4 to my notes based on those results. Finally, send a real message and confirm the hook's headlines reach the AI.

Once it runs, test it with reworded questions. Have a separate AI, one with no context on your work, write 20 rewordings of questions your notes answer, and count how often the right note lands in the top three. Have it write 20 unrelated questions too, and check that they get nothing or only a short list. Those two counts are what we used to choose the model and the rule for how much to hand over.

Wrapping up

My notes did not fail because there were too few of them. They failed because the AI did not go looking at the right moment, and when it did, the wording did not match. Loading every note at the start did not change that.

With RAG in place, every message is searched by meaning, a few headlines reach the AI before it answers, and nothing or only a short list arrives when the match is weak. The search runs on the Mac and costs nothing.

That changes what writing a note is for. A decision I write down now can come back the next time the situation does. The more of my own judgment I store, the more the AI has to draw on, and that is how RAG grows the AI without retraining anything.

If you keep giving your AI the same correction, check whether the note reaches the AI at the moment it matters, before you write another note. If it does not, the five steps above will put it in front of the AI on every message.

In this series, I show step by step how I store my own decisions and rules as records, and use RAG so the AI draws on them every time, with our own implementation and measurements.
If this was useful, please like and follow.

Our products are listed at https://www.eln.ne.jp/products

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.