將決策模型放進 Agent 迴圈，而非置於一旁

The cost nobody budgets for

Build an agent and count where the tokens go. A surprising share is not the work. It is the agent asking itself small questions.

Is this tool call safe to run. Does this document answer the question. Should this go to a human. Is this output in the right format. Did the last step actually succeed.

Each one is a yes, a pick, or a score. Each one currently costs a full generation: prompt in, reasoning out, parse the text, hope the format held. At a thousand of those a minute, the checking costs more than the task.

So teams stop checking. They drop the safety gate, skip the retrieval relevance test, let the agent run unguarded. Not because they decided the checks were unnecessary, but because the checks were too expensive to keep.

A different shape of model

A decision model answers without writing. Given the situation and a typed question, it returns a probability over the options in one forward pass.

Three shapes cover nearly everything an agent asks itself:

Nothing to parse. No format drift. No reasoning tokens to wait through.

Why calibration is the whole point

Accuracy alone does not let you build a gate. A gate needs a threshold, and a threshold needs probabilities that mean what they say.

If the model says 0.9 and is right nine times in ten, you can route everything above 0.9 automatically and send the rest to a human, and you can predict your error rate before shipping. If the model is accurate on average but its 0.9 means anything from 0.6 to 0.99, no threshold is safe and you are back to reading the output yourself.

This is why decision benchmarks score calibration, not just hit rate. Yes or no questions use Brier score. Choice questions score the probability mass on the correct option. Scales use expected score error. A model can top an accuracy table and still be useless as a gate.

What it changes in practice

Once checking is cheap you can check things you previously could not justify:

• Gate every tool call instead of the risky-looking ones
• Score every retrieved passage for relevance instead of trusting the top k
• Ask after every step whether the step succeeded, instead of discovering failure at the end
• Keep a confidence number on every decision so that the escalation rule is a threshold, not a guess

The architecture shift is small. The decision model sits inside the loop as a function call rather than beside it as another agent.

Reading the public boards

Decision models are now measured on public leaderboards such as S1MB, which currently covers 197 models across 137 benchmarks and ranks by aggregating ranks rather than averaging scores.

When comparing entries, read which column a claim quotes, the date it was read, and how many of the benchmarks the model was actually scored on. Most public boards mix self-reported and verified numbers.

Open-weight options exist in this category, so the approach is testable on your own traffic before it is a procurement decision. That is the right way to evaluate it: run your own decision log through a candidate and see whether the probabilities hold up at the threshold you intend to use.

A current reference point

If you want a concrete model to test this with, the S1MB english-v1 board currently lists Darwin-27B-ZTC-v9 first on Borda score at 90.38 across 137 benchmarks and 197 models, read on 20 October 2026. It is Apache-2.0, so you can run your own decision log through it rather than taking the board's word for it.

hf download FINAL-Bench/Darwin-27B-ZTC-v9 systemone_config.json --local-dir .
transformers serve FINAL-Bench/Darwin-27B-ZTC-v9 --decision-config systemone_config.json

The same board publishes a task-average column where that model is second at 72.20. Quote the column you mean.