Token Beggar 將編碼智能體子任務路由至較低成本模型是否划算？每組 63 次測試
Does routing coding agent subtasks to cheaper models pay off? A benchmark of 63 runs per arm
Token Beggar 的基準測試顯示，將部分編碼智能體子任務交由較低成本模型處理後，總成本由 $9.56 降至 $7.69，兩組品質評分均為 0.96。測試涵蓋 21 項任務、每組 63 次執行，兩組採用相同提示詞、可用工具及 Opus 5.5 high effort。作者以同一批 21 項任務調校路由，並在清除 bytecode 後重新評分兩組，因此結果僅描述這套設定，並非對其他項目的預測。
I wanted a straight answer to one question: if a coding agent hands its cheap subtasks to a smaller model, how much does a session really cost less, and does the result get worse?
The idea
Token Beggar is a plugin that adds a routing card to the session. The chat model grades each task into one of 7 classes and delegates bulk work (large reads, repeated edits, test loops) to a worker with its own context, at a cheaper model and lower effort. Design and root-cause work stays with the chat model. The chat model and effort are a ceiling, so nothing is routed above what you chose.
The benchmark
- 21 tasks on a small Python shop project, 3 per class, 3 runs each.
- Two arms per task, same prompt, same allowed tools, same chat model (Opus 5.5 at high effort).
- Code tasks: a hidden check plus the project's own tests. Answer tasks: a checklist scored by a blind judge that never sees the arm.
- Cost from the agent's JSON output, at list price.
| Class | Without | With | Cheaper |
|---|---|---|---|
| Multi-file feature | $2.47 | $1.58 | 1.56x |
| Mechanical edits | $1.11 | $0.80 | 1.4x |
| Reading | $0.91 | $0.75 | 1.21x |
| Design | $0.93 | $0.82 | 1.14x |
| Routine code | $1.47 | $1.32 | 1.12x |
| Bug root cause | $1.45 | $1.29 | 1.12x |
| Review | $1.21 | $1.13 | 1.07x |
| All 63 runs per arm | $9.56 | $7.69 | 1.24x |
Quality was 0.96 in both arms.
What I would not claim
Design, bug root cause and review never go to a worker, yet they came out 1.07x to 1.14x cheaper. That is run to run spread plus, I suspect, a shorter path the card gives the chat model. So I only quote multipliers of 1.15x or more in the plugin's footer. The routes were tuned on the same 21 tasks, so the table describes this setup and is not a forecast for yours.
A scoring bug worth knowing about
One debugging task looked like a regression in 5 of 6 runs with the plugin. The diffs were identical to the passing runs. The cause was a stale .pyc: the fix changed == to !=, which keeps the file size, and the bytecode written by the pre-fix test run still passed Python's mtime check. The scorer now clears bytecode before checking, and both arms were rescored. If you score code produced by agents, clear pycache first.
Also in the box
A compaction guard that writes a handoff note before an automatic compaction and defers it up to 3 times when files were edited after the last test run. The weekly model list refresh is a GitHub Action.
Repo, raw runs and the scorer: https://github.com/Arthur031221/Token-Beggar (MIT).
來源：DEV Community · dev.to