context-mode 評測：模型寫對腳本，才能減少 98% 上下文

I ran context-mode's own benchmark on my laptop and got its headline number to the byte: 314.9 KB of tool output in, 5,517 bytes into the context window. 98% saved.

Then I pointed one of the same benchmark scripts at the project's real git history. It reported that the most common commit type was "update". There is no such type.

Both results are true. The gap between them is the whole story.

The project

context-mode, by Mert Koseoğlu, is an MCP server plus hooks for coding agents: Claude Code, Codex, Cursor, Gemini CLI and more than a dozen others. It was on GitHub trending on Friday with 904 stars that day and just under 30,000 in total. It is source-available under the Elastic License 2.0; it launched as MIT and switched in March.

The idea is sound. Instead of letting a tool dump a 56 KB page snapshot into the conversation, the agent writes a small script, context-mode runs it in a subprocess, and only what the script prints comes back. Big outputs get indexed into a local SQLite full-text search, so the model can query them later. There is also a session memory that survives compaction, which I did not test.

The 98% opens the README, and it was the headline of a Hacker News thread that reached 570 points in February. I wanted to know what it counts.

How I checked

I did not install it. The npm postinstall and the server's startup both carry code that repairs Claude Code's plugin files, and I don't run that for a post. Instead I cloned the repo and loaded its executor and its search store straight from the TypeScript source, using Node 22's type stripping and a 12-line resolve hook, with HOME and TMPDIR pointed at a scratch folder.

No agent and no model in the loop. Just the project's code, run on files I chose.

What the 98% measures

The number comes from tests/ecosystem-benchmark.ts. It has 14 scenarios. Each one pairs a saved tool output (Context7 docs, a Playwright snapshot of Hacker News, 20 React issues, a test run, an nginx log) with a summary script written by hand for that file. Raw size is the file. Context size is whatever the script prints.

This is the parsing half of the git log scenario's script:

const lines = FILE_CONTENT.trim().split("\n");
const byType = {};
lines.forEach(l => {
  const type = l.split(" ")[4]?.replace(":", "") || "other";
  byType[type] = (byType[type] || 0) + 1;
});

So the 98% compares a file's size with what a script prints, where the script was written by someone who had already seen the file. It is a fair measure of how small a good summary can be. It does not show an agent doing the work.

I ran the benchmark through the project's own executor, and every row matched BENCHMARK.md to the byte.

What the count leaves out

In real use, the model writes that script, and the script is context too. It enters the conversation as the tool call. The current server also echoes it back: every ctx_execute_file reply starts with the file path and the code in a fenced block (up to 2,000 characters), then the output. The benchmark counts neither.

So I recounted the same 14 scenarios three ways, with the scripts' indentation stripped so I wasn't padding the total:

                            bytes    saved
raw tool output (315 KB)  322,419
stdout only (the README)    5,517    98.3%
+ echoed path and code     16,916    94.8%
+ the code in the call     27,504    91.5%

The scripts add up to 10,588 bytes, almost twice what they print. For the smallest fixture, a 400-byte list of network requests, the round trip costs 2,414 bytes, six times the thing it was meant to shrink. All of this is bytes, like the benchmark's own table; its token figures are bytes divided by four.

91.5% is still a big saving. My point is not that the number is fake. It is the best case, measured after the hard part was already done.

The script is the hard part

BENCHMARK.md says every fixture was captured from a real tool run. Some clearly were: the React issues match GitHub, number for number. The git log does not look captured. 30 of its 31 days have exactly five commits, and four of its five authors are Alice Johnson, Bob Martinez, Charlie Kim and Diana Chen. Every author has a two-word name.

That matters because of split(" ")[4]. It assumes the fifth word of each line is the commit type, which only holds when the author's name is exactly two words.

I ran the same script, unchanged, on context-mode's own history in the same line format (git log --format='%h %ad %an %s' --date=short): 2,234 commits, 203 KB. The 107-byte summary from the benchmark became 7,071 bytes, still a 96.5% saving, and its first entries were wrong:

By type: update:552, fix:168, Merge:125, docs:80, feat:66, ...

550 commits come from github-actions[bot], a one-word name, so the script read "update" out of "ci: update install stats". A three-word name turned the author's last name into a commit type. In all, 679 commits (30%) have an author name that is not two words. Reading the type from the subject instead, ci has 568 commits and fix, counting its scoped forms, has 663.

Nothing errored. The output looked like a summary. An agent reading it would have no reason to doubt it, because the 203 KB that would have exposed the mistake never entered its context. That is the trade: the savings come from the model not seeing the data, and so does the risk.

The HN thread got there first. One commenter, hereme888, noted that the benchmarks assume the model always writes the right extraction script. Asked about quality, the author said they "haven't run formal benchmarks on answer quality yet."

The safety net, tried

context-mode's answer to "what is lost" is its index. When you pass an intent and the output is over 5 KB, it chunks the output into SQLite FTS5 and replies with only the best matching sections. The full text stays searchable.

So I tried that path. I fetched 100 open React issues with gh issue list --json (303 KB), indexed them with the project's own store, and rebuilt the reply the way intentSearch in server.ts builds it, for the intent "ViewTransition bugs".

The reply was 1,667 bytes, a 99.4% saving. Eight of the hundred issues mention ViewTransition. None of their issue numbers appear in the reply. gh prints its JSON as one line, so the chunker split it by bytes: all five matches are titled "Output", and each preview is a 120-character slice of raw JSON, most from the middle of an issue body.

In fairness, the reply tells the model to call ctx_search for the full sections, and a careful model would. That is another round trip, though, and the first reply alone answers almost nothing. A model that piped the JSON through jq to keep only numbers and titles would have done far better. Same lesson: the quality lives in the code the model writes.

Where that leaves it

I'd use the idea without the package. When a command's output is big, have the agent write it to a file and run a script over it, and make the script print a few raw sample lines next to its counts, so a wrong parse shows itself. And judge tools like this on tasks, not bytes: did the agent get the right answer, and what did that cost, counting the code it wrote?

The limits of my check: I never ran context-mode inside an agent, so I can't say how often a real model writes a good script on the first try, or how the session memory holds up after compaction. The repo's GitHub issue tracker is turned off, so I couldn't read what users report there.

When your agent summarizes output you never see, what do you check before you trust the summary?