AI 智能體的 token 用在哪裏:Gemini 3.5 Flash-Lite 和 Gemini 3.1 Flash-Lite 測試 9 個 GitHub bug,各修復 1 個
Where an agent's tokens go: 9 real GitHub bugs, two Gemini Flash-Lite models, 1 fix each
Gemini 3.5 Flash-Lite 和 Gemini 3.1 Flash-Lite 各自處理 9 個 GitHub bug,合共呼叫模型 964 次,兩個模型都只修復 1 個 bug。按付費牌價計算,每個成功修復的 bug 分別花費 $0.561 和 $0.426;作者指出,未計入快取的工作流程計算器估算成本高出 2.4 倍和 1.8 倍。
I gave two Gemini Flash-Lite models the same 9 real bug reports from GitHub and let each one work in a loop: search, read a file, edit, run the tests, submit. Each model fixed one bug. Between them they made 964 model calls.
The tasks come from SWE-bench-Live (github.com/microsoft/SWE-bench-Live, MIT). I picked small pure-Python repos I could run on a Windows machine without Docker: attrs, faker and cfn-lint. A task only counts if the original tests fail before the fix and pass with the real fix applied. The agent never sees the new tests. When it submits, I apply them and run them. If they fail, it gets one more attempt and sees the tail of the failing output, like a CI log. Each attempt is capped at 30 steps.
The models were gemini-3.5-flash-lite and gemini-3.1-flash-lite, no thinking, on the free tier, so nothing was billed. The dollar figures below are token counts multiplied by the paid list prices ($0.30 / $0.03 cached / $2.50 per million tokens for 3.5, $0.25 / $0.025 / $1.50 for 3.1). Read them as what this trace would cost, not as a bill.
| 3.5 Flash-Lite | 3.1 Flash-Lite | |
|---|---|---|
| Bugs fixed | 1 of 9 (first try) | 1 of 9 (second try) |
| Attempts | 17 | 18 |
| Model calls | 483 | 481 |
| Input tokens | 4.29M | 3.00M |
| Output tokens | 16K | 24K |
| Input read from cache | 66% | 53% |
| Total at list price | $0.561 | $0.426 |
| Per task | $0.062 | $0.047 |
| Per fixed bug | $0.561 | $0.426 |
| Spent on tasks that never passed | 96% | 88% |
| Spent on second attempts | 49% | 46% |
Per task it is about 5 or 6 cents. Per fixed bug it is 43 to 56 cents, because one success has to carry all the failures. That gap is the whole idea of cost per successful task. It is also a single data point per model: the two models fixed different bugs, and a different task would have moved these numbers a lot.
Output was tiny, 16K and 24K tokens against millions of input. An agent loop sends the whole conversation back on every step. The first call of an attempt was about 1,060 tokens. The last call was 10,000 to 15,000. The cache helped, but fresh input was still 78% and 82% of the cost.
What did they do with their steps? 3.5 made 306 searches, 124 file reads, 33 test runs and 5 edits in 483 calls. 14 of its 17 attempts ran out of steps before it submitted anything. 3.1 searched less (160 times), edited more (50 edits) and submitted in 7 of 18 attempts, but 11 attempts still hit the cap, and only one of the submitted attempts passed the hidden tests. The only retry that paid off was 3.1 on attrs-1320: the second attempt saw the failing test output and fixed it.
I also put the measured numbers into three calculators on my site, to see which ones hold up on an agent loop.
- Importing the trace into the team API cost report gave $0.5611 and $0.4256, the same as my per-call sums. The trace format has no field for thinking tokens, latency or tool calls, so I keep those in a side file.
- The retry/fallback calculator, given about 2 attempts per task, 89% of tasks unresolved and the cached share, came to $0.5616 and $0.4258.
- The workflow calculator, which takes calls per task and tokens per call, said $1.33 and $0.78. That is 2.4 and 1.8 times too high. It has no cache input, so it bills every re-sent token at the full input price. For a loop that re-reads its context 50 times per task, that was the biggest error I found, and it is a gap in my own tool.
Things that make this weaker than it looks. Nine tasks, one fix each, on different bugs, so I can't say one model is better. The issues date from July 2024 to March 2025, so both models may have seen them. Google's pricing page lists context caching as unavailable on the free tier for 3.5 Flash-Lite, yet cache tokens were reported, and paid-tier hit rates could differ. The 30-step cap and my prompt probably cost both models some fixes. 3.1 hit its free daily limit of 500 requests after 9 of 12 tasks, so I left the other three out for both models. And 3.1 needed 14 API retries where 3.5 needed 1.
I built this to compare GPT-6.2 Sol and Gemini 4 Neon on the same tasks. I don't have an API budget for those yet, and Neon has no free tier, so I ran what I could. The harness has adapters for both, but I have only tested them against fake servers so far. There is also a write-up of cost per successful task if you want the definition I used.
來源:DEV Community · dev.to