Ajna 作者如何修復 harness，讓 gpt-oss:20b 完成原本需前沿模型處理的任務

A coding agent is a model plus a harness. We spend all our attention on the model. I spent mine on the harness — and a free 20B model started finishing tasks that used to need a frontier one.

I build Ajna, a terminal coding agent written in pure Python — no third-party packages, no build step, ~180ms cold start. It talks to Ollama, OpenAI, Anthropic, Groq, OpenRouter, and anything OpenAI-compatible.

But the part I actually spent time on isn't the agent loop. It's the layer that sits between the model and the tools and absorbs the mistakes small models make. This post is about why that layer exists, one counterintuitive thing I learned building it, and the benchmark that kept me honest about all of it.

The problem: cheap models drift

A frontier model calls the tools you give it, the way you described them. Small, free models — the kind you can run on Ollama's free tier or locally — don't. They drift. In testing I watched the same handful of failures over and over:

• They invent tool names. They'll call exec, bash, str_replace, read_file — none of which are the actual tool.
• They pass the wrong arguments: file_path instead of the expected name, the string "50" where an integer belongs, "true" for a boolean, plus extra keys the schema never defined.
• They corrupt files: replacement text whose lines start with +/-, as if they're writing a diff instead of the new content.
• They "finish" on a broken file — a write that leaves the file with an IndentationError, reported inside an otherwise cheerful "Created logstat.py (97 lines)".

Each of those costs a whole turn, and a weak model often can't recover from its own mess.

The move: don't make the model smarter, make the harness absorb it

Instead of prompting harder, I put a repair layer in front of the tools:

• Tool-name routing. ~60 invented names (exec, bash, str_replace, search, …) map onto the real tool.
• Argument aliasing + coercion. file_path, old_string, target_file, query and friends get renamed to what the schema expects; "50" and "true" are coerced to the right types; unknown arguments are dropped instead of raising.
• Diff-marker guard. Replacement text that looks like a diff is rejected before it touches the file — a real corruption I kept seeing.
• A syntax check after every write. Python, JSON, XML and TOML are parsed after the change. A write that leaves the file unparseable comes back as an error, not a footnote.
• Deletion notices. If an edit removes a function or class that existed before, the result says which — small models tend to replace when you asked them to add.
• Regex fallback. grep "total(" isn't valid regex; rather than failing the turn, it's searched for literally.

The measured effect on one task: a failing-test fix that took gpt-oss:120b 81 seconds and left a corrupted file before this layer existed now completes on gpt-oss:20b — the smallest model available — in 15 seconds, correctly.

The counterintuitive part: "go verify your work" made things worse

Here's the lesson I didn't expect.

The obvious way to make an agent more reliable is to tell the model to double-check itself: "when you're done, go and verify your work." I added exactly that and benchmarked it. On the bigger model it helped. On the smaller one it was a disaster:

Read as an instruction to keep working, a weak model edits files that were already correct. Across those v2 runs it dropped a key from a JSON file, produced JSON that no longer parsed, and deleted a test it had been told to keep. Mean time per task doubled, from 16s to 33s.

So v3 inverts it. The harness runs the checks — does every changed file still parse, is any removed name still referenced somewhere — and it speaks only when it has a specific fact to report. When the checks come back clean, the turn simply ends with nothing said. That narrowed the gap between the two models from 30 points to 10 — which is exactly what you want from scaffolding that's supposed to be model-agnostic.

The general lesson: scaffolding that asks a model to do more is not free. Only measurement tells you which way it cut.

The benchmark that kept me honest

Claims about an agent are worth what their measurement is worth, so Ajna ships with one. Ten tasks, each run in a fresh temp directory, each scored by running code rather than by reading what the model said: fix a bug the tests catch, wire a feature end to end, rename across files, edit JSON precisely, remove duplication without changing behaviour, repair a file that doesn't parse — and, crucially, answer a question without touching anything. An agent that edits when you only asked a question is a liability.

And I try to report the results honestly, including where they're unflattering:

• The v1 vs v3 numbers are within the noise of their sample sizes. v2→v3 on the small model (65%→80% at equal sampling) is a real gain; "v3 beats v1" is not a claim the data supports.
• Run-to-run reliability was the honest weak point: six runs of the same project-build prompt with gpt-oss:20b scored 34, 34, 32, 33, 27, and 0 out of 34. The zero shipped a file with an IndentationError — which the syntax guard had already detected but reported inside a "successful" message. Making an unparseable write a hard error killed that catastrophic mode; four runs since: 34, 32, 33, 33. Small sample, correspondingly small claim.
• Differential testing (feed every generated implementation the same malformed input, see where they disagree) surfaced four real defects — including two Windows encoding traps. Those now go into the system prompt as exact code to copy, because stating them as a judgement ("decide what to do on bad bytes") made a small model apply half of it and turn a working case into a crash. Concrete beat advisory, measurably.

None of this is a "my agent beats X" post. The honest takeaway is narrower and more useful: on small free models, most of the reliability you can buy comes from the harness, not from a better prompt — and the only way to know is to score by running code.

Try it

Pure Python, MIT, zero dependencies:

pip install ajna-cli
ajna --setup       # pick a provider (Ollama's free tier works), a key, a model
ajna "fix the failing test"

Repo, the full benchmark, and the numbers above: https://github.com/AbhiLabs421/Ajna

If you poke at the repair layer or the benchmark methodology, I'd genuinely like to hear where it's wrong. That's the whole point of shipping the measurement with the tool.