14 款 AI 編碼智能體的同任務基準測試呈現懸殊結果

In my last post, benchmarking a small local LLM exposed a bug in my own harness — the "weak model" was partly my code. That raised the obvious next question: how much does the tool around the model matter, measured across many tools?

So I ran 14 agent-type AI coding tools through the same task set, on the same ordinary PC (Windows 11, Ryzen 5 4500U — no GPU, nothing special), with two model backends:

BYOK: gemini-3.5-flash-lite via each user's own API key
Local LLM: qwen3.5:4b on Ollama

Scope: agent-type tools only — things that autonomously read files, edit files, and run commands. Chat assistants and subscription products (Copilot, Claude Code, Cursor, Devin) are out of scope.

What I measured

Real tasks: create a file, search/read and answer, perform a precise edit, diagnose+fix a failing test. Success rates count only the last 2 attempts per task, and results identical to baseline ("did nothing but claimed done") are marked failed.

BYOK results (gemini-3.5-flash-lite)

Tool	Passes
Cline (VS Code ext)	7/7 (100%)
Kilo Code, OpenHands, OpenCode, Aider	6/6 (100%)
Teaspoon IDE (standalone Electron)	6/7 (86%)
Zoo Code, Goose	5/6 (83%)
Continue	2/5 (40%)
SWE-agent, Void, Roo Code	0–1 attempts (0%)

Same model, same tasks — and the spread is 100% → 0%. The model is constant; the harness (prompt assembly, tool-call format, retry logic, context handling) is what varies.

Local LLM results (qwen3.5:4b, Ollama)

Tool	Passes
Cline	6/9 (67%)
Teaspoon IDE	6/9 (67%)
Void	2/3 (67%)
Aider	2/4 (50%)
Zoo Code, Kilo Code, Continue, OpenCode, Goose	~25%
Roo Code, OpenHands, Devika, Plandex	0%

The weak local model is a harsher judge: tools whose harnesses were tuned for strong cloud APIs fall apart (brittle tool-call formats, no retry on malformed output), while tools with tolerant parsing and explicit command protocols still complete tasks.

What the numbers actually show

"This model is good/bad" claims usually include the harness. Same backbone model, different success rate — that's tool-side design, not intelligence.
Small local models are the best harness test. A 4B on CPU has no slack; every weakness in the agent loop surfaces as a visible failure.
Dead ecosystems show up in measurements, not marketing. Roo Code's repo is archived; attempts got no response on either backend — the bench records it as "not evaluable" instead of pretending otherwise.
AI-vs-AI reviews add a second axis. I also had ChatGPT, Gemini, Claude, and Devin (SWE-2) score each tool's beginner-friendliness 0–5 by reading the measured data: Teaspoon IDE 3.8, Cline 3.5, everything else ≤2.8.

Caveats

Scores reflect the last two runs — small n by design (agent runs are slow on this hardware). Full per-attempt history is published per tool.
SWE-agent and Plandex hit environment constraints (Windows/autocrlf, no native build) — recorded as not evaluable, not failed.
I'm the author of one tool in the table (Teaspoon IDE). The measurement protocol is identical for all tools and the raw logs are public — judge for yourself.