何時值得為單次請求付費使用推理模型

A reasoning model spends extra tokens before it answers. Send every request there and you pay latency plus a larger bill for lookups a standard model already handles. Skip thinking entirely and you give up the accuracy gain on hard problems.

A standard model answers in one pass. A reasoning model is trained to run an internal chain of thought first, exploring, checking, and revising before it commits. That extra work is test-time compute: the model stays fixed and spends more effort while answering. On math, code, and multi-step logic, that thinking raises accuracy. DeepSeek made the trade concrete, with strong reasoning from heavier inference, using reinforcement learning to auto-generate step-by-step reasoning data at scale.

Before that thinking was the default, chain-of-thought prompting asked for the steps in the prompt. A 2022 Google Brain paper by Wei et al. showed that this lifts accuracy on math, logic, and planning. Few-shot chain of thought supplies worked examples. Zero-shot chain of thought adds a cue such as "let's think step by step", which is cheaper and often enough on a capable model. Self-consistency samples several chains and keeps the majority answer, trading more tokens for a further bump on hard problems. On a reasoning model, that cue matters less, because the long chain is already built in.

A fluent chain can still be wrong. An AI hallucination is a false or fabricated claim stated as fact. It falls out of how these models generate text. A plausible chain can sit beside a wrong result, so the chain does not make the output trustworthy. I would pay for the tokens only when I can name what the extra thinking is buying.

What should you check before you pay?

I use three checks, in order. Fail any one and the request stays on the standard model.

First, can you tell which answer worked? Dan Klein argues that a simple picture of a reasoning model is trying several tactics and keeping the one that worked. That helps on math: once the right tactic is found, the proof goes through. I would give those attempts something to land on: a unit test, a schema, a calculator, or a known label. If you cannot separate a good answer from a fluent one, a longer chain can just give you a longer wrong answer.

Second, does the latency budget allow a slow answer? More test-time compute means a slower response. A chat reply the user is waiting on may have no room for a long internal chain. A batch job, a CI check, or a nightly report usually can.

Third, is the token cost of thinking smaller than the cost of being wrong on this request? Klein's warning on when to skip that spend is blunt.

But then humans take this even further. there's like a chip of rock flying at your face, you just blink, you don't stop and think, huh, it would be bad if this rock hit my eye, I can move my head, I can close my eye, you don't do that, you just blink. And so as humans, we use different amounts of computation and meta-computation based on the situation we're in. Reasoning is a very particular kind of use of test-time computation, which works for some problems and absolutely is not the right solution for others

Dan Klein, co-founder and CTO of Scaled Cognition, on Chain of Thought episode 54

He also argues that once you have done a task a couple of times, planning should become memory. I would cache a known procedure, or call a standard model, instead of re-solving that task from scratch on every request.

There may be a cheaper step between the two models; measure it. On a capable standard model I would try zero-shot chain of thought first, then self-consistency if the checker still fails and I can afford several samples. I would turn on the reasoning model only for the slice that still fails those steps and still passes the three checks.

How do you measure the difference on your tasks?

Public scores are a shortlist. The ship decision comes from tasks you control. An AI benchmark is a fixed test with a shared scoring method, such as MMLU for knowledge or HumanEval for code. A model that leads a leaderboard can still fail on your data, your formats, and your edge cases, and benchmark items can leak into training data, which inflates the score. Pick candidate models from those scores. Decide the route by scoring them yourself.

Dan Klein argues that a bad metric lets you talk yourself into progress you do not have. He contrasts a system that is right on most scenarios, but wrong often enough to break a customer action such as booking a flight, with a system that covers fewer scenarios and gets those right every time. The second one is what you can ship. He also argues that every team has to decide what it needs and check that the metric captures that, because it is easy to believe a system will meet the need when it will not.

I would pull a small set from real traffic: hard multi-step items, simple lookups, and routine cases you have already solved. For each item, run the standard model and the reasoning model. Record whether the checker passed, how many tokens came back, and whether latency beat the budget. The reasoning model earns that class of request only where accuracy rises enough to justify the extra tokens and the slower replies. On lookups, expect the sheet to show overkill, and keep those calls off the reasoning route.

If you cannot build a checker, leave those items out of the automatic comparison. Sample them by hand. Treat a confident chain as untrusted text.

What does a routing sketch look like?

This sketch shows the rule and the scorecard. It is my illustration, not a design any guest or company uses, and it calls no vendor SDK.

# Sketch only. A routing rule plus a private scorecard.

def route(task, budgets):
    if not task.checkable:
        return "standard"
    if budgets.latency_ms < task.reasoning_p95_ms:
        return "standard"
    if task.expected_thinking_tokens > budgets.max_thinking_tokens:
        return "standard"
    return "reasoning"

def scorecard(items, run_standard, run_reasoning):
    rows = []
    for item in items:
        a, b = run_standard(item), run_reasoning(item)
        rows.append({
            "id": item.id,
            "standard_ok": item.check(a.text),
            "reasoning_ok": item.check(b.text),
            "token_delta": b.tokens - a.tokens,
            "latency_delta_ms": b.latency_ms - a.latency_ms,
        })
    return rows

Read the rows before you point route at live traffic. Where reasoning_ok is true, standard_ok is false, and both deltas fit the budget, that class can go to the reasoning model. Where both checks pass, keep the standard model. Where item.check cannot run, that row does not belong in this loop.

Checklist

• Pay for a reasoning model only when a checker exists, the latency budget covers a slow answer, and the extra thinking tokens cost less than being wrong.
• Try zero-shot chain of thought, then self-consistency, on a standard model before you make the reasoning model the default.
• Shortlist with public benchmarks, then score both models on your items for pass rate, tokens, and latency.
• Keep lookups, routine work, and uncheckable prose on the standard model.
• Re-score when the task mix changes. A model that thinks well on math can be the wrong spend on the rest of the traffic.

More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.