Arize 與 Fireworks 的 2,400 次測試顯示，每 token 成本最低不代表成功完成任務成本最低

A model with a 33% pass rate turned out to be the cheapest way to finish a task in a recent 10-model benchmark. The same benchmark shows why that result is less of a free lunch than it looks. Here is the metric behind it and how to compute it on your own prompts.

The metric

Cost per successful task = everything the model spent across every attempt (failed, retried, timed out) divided by the number of attempts that passed.

Per-token price cannot see retries, malformed outputs or runs that never finish. This metric sees all of them.

The numbers

Arize and Fireworks ran 10 models on 40 real command-line tasks from Terminal-Bench, 6 trials each, 2,400 runs. The graded outcome is the task's own test script. A selection:

Source: Arize and Fireworks, cost per successful task.

Three things to read from it:

1. Retries are a tax you pay in dollars. gpt-oss-120b needs about three attempts per success, and is still roughly 12 times cheaper per finished task than GPT-5.5 ($0.054 against $0.636).
2. A cheap model is not always a cheap result. gemini-3.5-flash costs $0.288 an attempt but passes 23% of the time: $1.233 per success, more than twenty times gpt-oss-120b's.
3. Coverage is the catch. Counting only tasks a model solved reliably (4 of 6 trials or better), gpt-oss-120b covers 8 of 40, GPT-5.5 covers 25 and Kimi K3 covers 26. Part of the cheap model's advantage is that it only wins the tasks it can win. Swap it in wholesale and the bill and the feature set both shrink.

By difficulty, price tier did not predict who wins the easy tasks: GPT-5.5 passed 69% of them, Kimi K3 (also frontier-priced) 100%, and gpt-oss-120b 65% at roughly a twenty-fourth of the cost per attempt. On hard tasks the gap was large (GPT-5.5 51% against 14% for gpt-oss-120b). With 6 trials per cell, the source puts the pass rates at about plus or minus 6 points, so read the difficulty split as a direction, not a decimal.

What does and does not transfer

These are agentic terminal tasks, not classification or extraction, so do not read the dollar figures as your numbers. The method transfers.

Run it on your own prompts (an afternoon)

1. Take 100 real inputs and the output you would accept for each. Real traffic, not a demo set.
2. Write the pass rule before you run anything: exact match, a JSON schema plus field checks, or a rubric. Set the bar (say 90%).
3. Run your current model and two or three cheaper candidates. Record the cost of every call from the provider's usage field, including calls that were retried or rejected by your validator.
4. For each model compute: passes / total, total spend / passes, and the share of calls that needed a retry.
5. Throw out any candidate under your bar. Of the rest, the lowest cost per success wins. If that is your current model, you have learned that for free.
6. Re-run when a price drops. A cheaper candidate you rejected last quarter may clear the bar at a lower price now, and prices move monthly.

The headline of that benchmark is the one worth remembering: the label (open or closed, frontier or budget) was the wrong axis. Cost per successful task, on your task, was the right one.