我的 LLM 路由器多花 8 倍成本，準確率卻與廉價模型相同

I built the LLM model routing strategy almost everyone reaches for first: ask the cheap model to
answer, have it report how confident it is, and send the question to a stronger model whenever
that confidence drops below a threshold. Thirty tasks, three strategies, real local models, real
published pricing applied to the real token counts. The router escalated 14 of the 30 tasks to
the strong model and ended up with exactly the same accuracy as never routing at all, 43.3%
either way, while spending 8.2 times more money to get there.

My first read of that number was "routing did nothing here." That read was wrong, and the only
reason I know it's wrong is that I went back and checked every task individually instead of
stopping at the headline. Two real, opposite effects happened. They canceled out almost exactly,
which is a coincidence of this specific run, not a law the router obeys.

Proof block. The setup, before any of the findings:

• 30 tasks (10 easy, 10 medium, 10 hard), each graded by a programmatic rule (exact or substring match against a known answer), no subjective scoring.
• Models: llama3.2:1b (cheap tier) and llama3.1:8b (strong tier), via local Ollama 0.34.2.
• Every call: temperature 0, fixed seed, zero retries.
• Illustrative cost: real token counts from each call, priced at DeepSeek V4.1 Flash's published peak-hour, uncached-input API rate ($0.30 / $1.20 per million input/output tokens; off-peak drops to $0.15/$0.60) for the cheap tier and GPT-5.6 Sol's published short-context rate ($4.00 / $20.00 per million tokens) for the strong tier. Local execution itself cost $0.
• Full raw data, code, and a second verification pass: github.com/priyansh18/confidence-router-benchmark.

TL;DR. What actually came out of checking the numbers instead of trusting the headline:

• Routed accuracy tied always-cheap accuracy exactly (43.3%, 13/30), while costing 8.2x more than always-cheap. Always-strong reached 66.7% accuracy, and routed still cost 3.2x less than that.
• Only 3 of the router's 14 escalations actually changed an answer from wrong to right. 8 were pure waste (the cheap model already had it right); 3 escalated into a task neither model could solve.
• 14 of 16 non-escalated tasks were wrong answers the router let through at reported confidence 70 or higher. Self-reported confidence and actual correctness barely correlate here.
• A confound I did not plan for: asking the cheap model to also report its confidence, in the same call as the task, changed its actual task answer on 5 of those 16 tasks, four of them from right to wrong. That alone cost 3 net correct answers, independent of any escalation decision.
• The escalations that worked (+3 net) and the confidence-instruction side effect (-3 net) landed the final number back on the cheap model's own baseline. Different task mix, different result.

The question this is actually testing: does a model's own self-reported confidence predict
whether its answer is correct, well enough to use as a routing signal? A falsifiable claim, not a
vibe. A separate, closer question the write-up below surfaces almost by accident: does asking for
that confidence score change the answer itself?

LLM model routing, the version everyone tries first

LLM model routing means sending each request to the cheapest model that can actually handle it,
and only paying for a stronger model when the task needs it. The published version that works is
RouteLLM (Ong et al., 2024, UC Berkeley/LMSYS): a router trained on real preference data, human and
LLM-judge comparisons of which model actually answered better, achieving over 85% cost reduction
on MT Bench while keeping 95% of GPT-4's quality, at best sending only 14% of traffic to the
strong model (GPT-4 Turbo vs. Mixtral 8x7B, their best augmented-data router).

That takes a labeled dataset and a training step most people building an agent on a Tuesday don't
have. The version that's actually quick to try is the one in this article: skip the training,
just ask the cheap model how sure it is, and escalate when it says it isn't.

The 30 tasks span the kind of decisions a production agent harness routes constantly: factual
lookups and unit conversion (easy), log-field extraction and multi-step cost arithmetic (medium),
and multi-constraint reasoning with a distractor built in (hard). Every task has a programmatic
check() function, exact or substring match against a known answer, so grading never depends on
my own judgment call.

The routing rule under test:

ESCALATE_BELOW = 70

def route(task, cheap_model, strong_model):
    routed_prompt = task.prompt + (
        "\n\nAfter your answer, on a new line write exactly: "
        "CONFIDENCE: <a number 0-100 for how sure you are>."
    )
    answer = call(cheap_model, routed_prompt)
    confidence = parse_confidence(answer)

    if confidence is None or confidence < ESCALATE_BELOW:
        return call(strong_model, task.prompt)  # escalate, plain prompt
    return strip_confidence_line(answer)

Cost for an escalated task is the cheap call's cost plus the strong call's cost. Cost for a
non-escalated task is just the cheap call. I ran all three conditions, always-cheap (plain
prompt, no confidence instruction), always-strong (plain prompt), and routed, against every one
of the 30 tasks.

Results before the explanation

Image by author. Pricing: DeepSeek V4.1 Flash ($0.30/$1.20 per M tokens) for the cheap tier, GPT-5.6 Sol ($4/$20 per M tokens) for the strong tier, applied to real local token counts.

The router escalated 14 of 30 tasks (46.7%). Here is what those 14 escalations actually bought:

• 8 wasted. The cheap model's plain answer would already have been correct. The escalation changed nothing except the bill.
• 3 helped. The cheap model was wrong; the strong model fixed it. This is the one thing the router is supposed to do, and it did it exactly 3 times.
• 3 still wrong. Both models missed it. A genuinely hard task, not a routing failure.

So escalation itself was mildly useful on net: +3 correct answers, and it never broke an answer
that was already right (confirmed, all 8 wasted-bucket tasks stayed correct on the strong model
too). If that were the whole story, routed accuracy should have landed at 16/30, not 13/30. It
didn't. Something else happened.

Where the other 3 correct answers went

The 16 tasks that did not escalate were not simply "kept the cheap model's plain answer." Each
one used a second call: the task prompt plus the added confidence instruction. I'd assumed, going
in, that this second call would answer the task the same way the plain call did, it's the same
model, same temperature, same seed, just one more line of instruction asking it to also grade
itself. That assumption was wrong on 5 of those 16 tasks.

Image by author, computed directly from raw_results.jsonl.

Four tasks that the plain cheap model answered correctly got answered wrong once the
confidence instruction was attached to the same prompt. One task flipped the other way, wrong
without the instruction, right with it. Net: minus three correct answers among the non-escalated
set, purely from adding one line asking the model to grade its own certainty. Plain-prompt
accuracy on those exact 16 tasks would have been 31.3% (5/16); the actual routed-prompt accuracy
on them was 12.5% (2/16).

One of the four right-to-wrong flips was the simplest task in the whole set: "what is 47 + 58?"
Asked plainly, the cheap model answered 105, correct. Asked the identical question with one extra
sentence attached, report a confidence score afterward, it answered 95, wrong, and reported 80%
confidence in it. Nothing about the math changed. The only difference between the two calls was
one trailing instruction the model never even needed to act on until after it had already
answered. For a 1.2-billion-parameter model, that was apparently enough to disturb the arithmetic
itself, not just the metadata wrapped around it.

A second, costlier case, task h5, shows the same failure mode on harder math: "a classifier has
95% accuracy on 80% of traffic and 60% accuracy on the other 20%, what's the overall accuracy?"
The correct answer is 88%. The cheap model wrote out the right formula, (0.8 x 95%) + (0.2 x
60%), under the routed prompt, then slipped on the arithmetic and wrote 76.8% instead of 88%,
reporting 100% confidence in that wrong number. This one wasn't a flip (the plain-prompt
cheap model missed it too), but it's the sharpest example in the set of what "confidently wrong"
actually looks like: a correct method, a blown calculation, and total certainty in the output.

I did not design this run to isolate that effect, which means I can't yet say how general it is
across model sizes or instruction phrasing. What I can say is that it happened, on a clean,
deterministic, reproducible setup, and it happened to land in the opposite direction of the
escalation gain, which is the only reason the headline number looked like nothing changed. A
router implementation detail, whether you ask for a confidence score inline versus in a separate
call, measurably changed what was being routed, before the routing decision even mattered. If you
are building something like this, that is worth a dedicated check of its own, not an assumption.

What this means if you're building a router today

Self-reported confidence is not a usable routing signal by default. Here it failed in three
separate ways at once: wasting money re-asking questions it already had right, silently keeping
wrong answers it was sure about (14 of 16 non-escalated tasks), and degrading accuracy as a side
effect of the instruction used to elicit it. Any one of those would be worth fixing before
shipping a router built this way. Getting all three at once is a strong argument against the
one-line version of this idea, not a reason to abandon routing altogether.

Test the full implementation, not just the escalation logic. The part I expected to be weak
here (confidence threshold picking) was weak, but not the only problem. The part I didn't expect
to matter (how the confidence instruction is delivered) cost as much accuracy as the escalation
logic gained back. If I'd only checked the top-line numbers, I would have reported "routing had no
effect" and missed both real findings underneath it.

If routing cost actually matters for your workload, build toward RouteLLM's approach, not this
one. A router trained on real preference comparisons between model outputs is the proven path.
If that's not feasible yet, this run's own comparison says something useful on its own: on this
task mix, always-cheap strictly dominated the naive router, identical accuracy, at about an
eighth of the cost (8.2x less). A null router can beat a bad one.

Resources

Everything here, the 30 tasks, the exact prompts, every raw model response, the token counts and
cost math, the three annotated failure cases, and the chart generator, is public and reproducible:
github.com/priyansh18/confidence-router-benchmark. Deterministic by design; a clean re-run
reproduced every accuracy number exactly when I checked it mid-project, which is what first made
the discrepancy in this article worth chasing down instead of writing off as noise.

If you're building agent infrastructure and want a second pair of eyes on a cost or reliability
problem before it ships, I take on a limited number of consulting calls through
Topmate.

Parts of this article's drafting were assisted by an AI writing tool; the experiment, the data,
the repo, and the analysis are my own.

One real question, not a generic one: if you've shipped a confidence-based or heuristic
router in production, did you ever check whether the signal you used to decide "escalate or not"
actually correlated with correctness on your own traffic, or did it just look reasonable at
design time the way mine did before I ran this?