1.5B 模型在程式碼審核中是分流訊號，而非裁決者

I run an auditor that reads trading-strategy repos while I sleep. A 1.5B model is part of it, and for a long time I described that part wrong.

The tool runs five deterministic checks over the source, then hands each finding to a local model with exactly six lines of context and one yes/no question: is this actually true here? Nothing leaves the machine. No API key, no upload, no account. The model is Qwen2.5-Coder-1.5B-Instruct Q4_K_M, loaded through llama.cpp, on a laptop CPU.

The interesting part is not that it works. It is where it stops working, and what I changed once I measured that.

The 48-second constraint

Four model checks on the bundled sample take about 48 to 50 seconds on this laptop. That is roughly twelve seconds per question. The deterministic pass, --no-llm, takes under a second.

Twelve seconds per question is the fact that shaped everything else. You cannot put that inside a loop over every file in a repository. You cannot put it behind a save button. What you can do is run it at night, or while you make coffee, and then read a report that is already sorted.

Latency also removes a whole class of bad ideas for free. I stopped considering "ask the model to look around and tell me what seems off" not because it is philosophically wrong but because it costs twelve seconds and returns a paragraph, and I do not want a paragraph at that price. I want a pointer.

One question, six lines, three answers

The model never sees the repository. It sees the finding's evidence line plus five lines around it, and it can answer YES, NO, or UNSURE. It is never asked to name a file, never asked for a line number, never asked for a summary.

That last restriction came from a failure. In an early build I asked it to summarise the file. The output was fluent, specific, and wrong about which function did what — the kind of answer that is expensive to read because you have to verify every clause. A checker that points at line 9 and a model that confirms or denies a claim about line 9 is a pipeline. A model that writes prose about code is a different product, and not one a 1.5B can deliver.

So the division is blunt:

• Locations come from code, never from the model.
• The model answers one narrow question per finding, and "UNSURE" is a real answer, not a failure.
• The report prints the evidence line verbatim, so the reason to read a finding is never the model's prose.

The instability I could not engineer away

Here is the measurement that changed my design.

I asked Gemma 2 2B IT Q4_K_M (temperature 0.2) the same yes/no question about the same code, three times, in isolated contexts. The answers were NO, NO, YES.

Same weights, same prompt, same evidence line. Two of three said the cost bug was not there. It was there; the checker had already printed the ledger line with no fee term in it.

I also run both models and print their answers side by side, as a confidence signal. Two runs of the same audit gave 3 of 4 agreement and then 4 of 4. It would be tidy to call that sampling variance, but it is not: the code changed between the runs — I moved the context window to the evidence line — so the difference is confounded. I would rather write that down than round it into a nicer story.

Both runs are committed in the repository, and so are the three re-asks. That is the only claim I am willing to make: at 2B, on this task, the verdict is not a stable measurement. It is a hint about where to look, from a reader who has read six lines.

Once you accept that, the design stops fighting the model.

Five rules that follow

1. The checker owns the finding. Every path:line in the report comes from code that matched a pattern. The model cannot create a finding, move one, or delete one.
2. One question per finding. Narrow scope keeps a 1.5B inside the task. Broad scope turns it into a confident narrator.
3. The model is allowed to say nothing. "UNSURE" and "nothing to add" are valid outputs, and the report still finishes without them.
4. Agreement is a hint, not a vote. Two models agreeing means neither disputed the checker. It does not mean the finding is real, and the report never prints a confidence score I cannot compute.
5. Absence checks print what they searched. "No fee term in this file" is a weaker statement than "no match for fee, commission, slippage, or spread across 214 lines". For negative findings the evidence is the search, so the search has to be in the output.

The blind spot is written in fixed text, not generated: the tool does not run the strategy, does not see your data, and cannot tell you whether the edge is real. All it can do is point at the places where a backtest usually lies and ask a small model whether the pointer is correct.

What I have not measured yet

The honest gap is the threshold. Right now a finding either fires or it does not, and the model's answer is read by a human. If I ever wanted to automate the decision — skip the ones the model denies, say — I would need two numbers I do not have:

• the margin around the cutoff, meaning the distribution of scores for true findings versus near-misses, not three anecdotes;
• the flip rate under prompt shift, meaning the same question with a quoted line added or the field order changed.

Both are cheap to measure and both are missing from this post. That is the difference between a demo and something I would let run unattended on a repository I care about. I would rather say that here than discover it in production.

When to skip the model entirely

--no-llm runs the deterministic checks only, instantly, and it is the right mode more often than you would think:

• when you need the same output twice, byte for byte;
• when the repository is small enough that reading the report is faster than waiting;
• when you are about to argue with someone about a number and cannot afford a witness who changes its mind.

The model is worth its twelve seconds when the finding is ambiguous and you want a second reader at 3am. It is not worth it when you already know what to look at.

Try it

pip install -e .
slothaudit audit samples/broken_lookahead            # ~50s, four model checks
slothaudit audit samples/broken_lookahead --compare   # both models, one report
slothaudit audit samples/broken_lookahead --no-llm    # deterministic only, instant
python3 -m slothaudit audit samples/broken_lookahead  # works without install

A clean repository gets a deliberately un-reassuring answer: no v0 check fired; that means this run had nothing to point at, not that the strategy survives an audit.

Code: github.com/rytjy/slothaudit

Disclosure: I used an AI assistant to help draft and edit this post. The measurements, the code, and the command output are mine, and the output above is pasted verbatim from runs on this machine.