TokenCap 撤回 v1.6 的 12.2× 基準宣稱，列出五項評測錯誤

TokenCap v1.6 shipped with a line we liked a lot: "12.2× fewer tokens, same answer." It came from benchmarks/run.js. It was wrong in five separate ways, and it is now retracted in docs/HONEST-NUMBERS.md.

This post is the list, and what replaced it.

The five problems

1. "Same answer" was never measured. The harness printed the phrase. It called no model and checked no answer. The words were a label, not a result.

2. The naive baseline counted TokenCap's own output. The capsule was written to TOKENCAP.md inside the repository. The naive side then read the repository, capsule included. Every naive figure carried the capsule's full size, which inflated the baseline we were claiming to beat.

3. One task matched every file. Keywords were matched against the absolute path. On Windows the temp path starts with C:\Users\..., and user was one of the keywords. So one "naive" read was simply the whole repository.

4. TokenCap's side was not task-specific. One capsule per repository was reused for all five tasks. The 15 data points were really three measurements.

5. The two headline figures used different aggregations. 12.2× was the mean of per-task ratios. The 94% figure came from the ratio of the averages. Two numbers, two methods, side by side.

None of these needs bad intent. Each one is a shortcut that looks fine until someone reads the harness carefully.

What the corrected number looked like

With the baseline fixed, the small repository's capsule was 8,275 tokens. The naive read averaged 3,880 tokens. The capsule was about twice the size of what it was supposed to save.

That is exactly the kind of result a benchmark exists to surface. The old one could not surface it.

What replaced it

benchmarks/fair/ scores both sides against ground truth instead of against each other's size:

• Each task is a real commit from one of nine repositories (JavaScript, TypeScript, Vue, Python, Go, Rust, Java).
• The task text is the commit subject. The ground truth is the set of non-test source files the commit changed.
• The repository is checked out at the commit's parent, so the answer is not in the tree yet.
• Commits are selected mechanically by a rule recorded in benchmarks/fair/tasks.json. 50 were used while changing retrieval and packing code. 50 were held out.
• Both sides are scored on tokens and on whether the needed files are present. A smaller pack only counts if it still holds what the task needs.

Results across all 100 tasks:

Note the first row. Reading every matching file finds the needed files more often than the pack does. We publish that next to the saving, not under it.

What changed in how we publish

• A CI job runs the benchmark and fails when needed-file recall, edit-line coverage, or the median saving drops below the thresholds in tasks.json.
• The limits sit next to the numbers: no model is called, grepping everything finds more, small repositories gain little, and the median is quoted because the mean (34.2×) is pulled up by the large repository.
• The old script and its result file stay in benchmarks/, for the record only.
• Before the v3.1 retrieval changes, the first three repositories scored 25% needed-file recall on this same benchmark. That earlier result set is kept in benchmarks/fair/results/ as well.

That 25% is the number I would point to if someone asked whether the rebuild was worth it. The old benchmark could never have shown it, because it never asked whether the right files were in the context at all.

The full table and limits list are at tokencap.vansharora.app