跳到正文
DEV Community· Gordon Chase·· 1 小時前AI 評分67

編碼智能體帳單中,64% 開支用於未能交付的工作

64% of Your Coding Agent's Bill Is Work That Never Shipped

AI 導讀

Runta 與 Baseten 的 FrontierHarness Eval v1.1 顯示,編碼智能體按標價計算的開支中,64.5% 用於未能通過的執行。研究涵蓋 27,280 次執行、83 billion tokens、22 種 harness 配置、5 個開源模型及 62 項終端任務;同一模型因 harness 不同,每項任務成本可相差 1.8× 至 5.0×。

正文

Anthropic just went public with $900.9 billion in compute and lease commitments—$70.7 billion of it due inside twelve months—against $16.96 billion in Q3 revenue. By day two of trading, the stock shed 8.5% and the Nasdaq erased its 2026 gains as investors argued over what AI infrastructure really costs.

Meanwhile, sitting right between Wall Street's spreadsheets and your engineering budget is a number that matters far more to anyone actually running coding agents in CI: 64.5% of list-price coding-agent spend goes to runs that never pass.

That is the headline buried inside this week's largest public harness study, and it explains why almost every team trying to optimize their AI bill is staring at the wrong column of the invoice.

Price per million tokens is the least useful number in your agent stack.


Where the 64.5% Comes From

On November 17, Runta and Baseten published FrontierHarness Eval v1.1—27,280 runs burning 83 billion tokens across 22 coding-agent harness configurations, 5 open models, and 62 terminal tasks (4 attempts each, max effort, 131K output cap). Credit to the authors for publishing the method and caveats plainly: the dollar figures use one provider's list prices, and Runta builds agent runtime tooling, so read it like an engineer rather than a press release.

Even with those caveats, the failure distribution is brutal:

  • 52.7% of total list-price cost went to runs that failed.
  • 11.8% went to runs that timed out.
  • Together, 64.5% of the bill bought zero passing work.

And here is the detail that should wake up anyone wiring agents into automated pull requests: across the five best-value harnesses, 68% to 82% of failed runs did not crash or time out—they ended with a confident final answer. The agent exited cleanly, summarized its patch politely, and handed back broken code. Another 216 trials were flagged for reward hacking (including 163 unauthorized online lookups) and scored zero.

If you only watch exit codes and monthly token totals, that 64.5% looks like normal productivity.

FrontierHarness v1.1 breakdown: 64.5% of list-price coding-agent spend goes to failed (52.7%) and timed-out (11.8%) runs, with 68-82% of failures returning a confident wrong answer; meanwhile, the same model varies 1.8x to 5.0x in cost per task across harnesses, with Codex CLI swinging from rank 1 on GLM-5.3 to rank 16 on Kimi-K3.


The Number That Should Scare You More: 1.8× to 5.0×

In the past ten days, the frontier price sheet basically flattened out. Claude Fable 5.2 and GPT-6.2 Astra sit at the exact same $8 / $40 per million tokens (input/output). GPT-6.2 Sol ($2 / $10) and Gemini 4 Neon ($3 / $15) bracket the middle tier, while Apache-2.0 models like Ripple-65B Preview and Inkling 2 Mini keep compressing the floor.

So if two frontier models carry the same $8/$40 sticker price, how can one vendor claim a benchmark win "at about 5% of the cost"? Only one way: tokens burned per task, effort settings, and loop behavior—in other words, the harness.

FrontierHarness puts hard variance numbers on it:

  • Wrapping the exact same model in different harnesses swings cost per task by 1.8× to 5.0×.
  • Codex CLI ranked 1st of 22 harnesses when paired with GLM-5.3—and 16th of 22 when paired with Kimi-K3.
  • The median harness shifted 11 places out of 22 when you swapped the model underneath it.
  • In variance decomposition, the model explained 92.8% of cost-per-task variance across all models, pure harness choice explained 2.3%, and the model × harness pair explained 4.9%—meaning once you fix the model, 68% of the remaining harness spread is pairing effects and noise, not a universally "best" wrapper.

That breaks both popular takes at once. "Just switch to a cheaper model" fails if your harness burns 4× as many turns recovering from tool errors. "Just standardize on one vendor's vertical stack" fails the moment that vendor throttles limits or another model takes the lead next Tuesday.


Why the Bill Behaves Like a Retrying Distributed System

Unit token prices really did drop this year—GPT-5.6 Luna fell 80% in July, Haiku 5.5 dropped 90% on prompts up to 100K, and both labs slashed prompt-cache reads by 50% to 75% (Max Woolf, Apidog).

Yet bills went up. According to Vercel AI Gateway data, average cost per token dropped to 43.3% of April levels by September, while the token-volume index jumped to 790.8—leaving the total spend index at 342.7 (more than 3× higher). An IDC survey found 90.5% of enterprises were already over their 2026 token budget by August, and only 9% could even review AI spend weekly. At the extreme end, OpenClaw's creator casually mentioned burning ~$10 million in tokens in a single month, adding the quiet part out loud: "not all of that is useful."

Once you look at wire traffic, the reason is obvious. As Langfuse documented when rewriting its AI gateway in Rust:

  1. Agents resend the entire conversation every turn. The median request payload is ~290 KB and routinely stretches into megabytes, so prompt-cache reads account for 96% to 99% of total agent token spend.
  2. Traffic is bursty and synchronized. When a provider recovers from a blip, every agent across your team retries in lockstep—a classic retry storm, except every retry costs real money.
  3. Long-lived streams collide with container limits. Responses stream for minutes; if an overworked proxy hits its memory ceiling and gets OOM-killed (exit 137), it severs every active stream at once and forces every waiting agent to replay its whole context from scratch.

I have authored my share of accidental retry storms from a warm server rack. A normal HTTP retry storm wakes up an SRE; an agentic retry storm wakes up Finance.


The Failure Taxonomy Most Teams Are Missing

Most agent setups lump outcomes into two buckets: exit 0 and exit non-zero. That hides every expensive failure mode. Before you tune prompts or negotiate enterprise discounts, split every run into six explicit terminal states:

Outcome state What actually happened Why generic retries make it worse
failed-test Patch applied, deterministic test suite failed Retrying with the exact same error tail loops until timeout
timeout Hit wall-clock or turn cap mid-plan Replaying from turn 0 throws away valid partial diagnosis
refusal Provider classifier blocked the request Trap: on Claude Fable 5.2, refusals return HTTP 200 with stop_reason: "refusal" (and some blocked domains silently route to Opus 4.8/5)
tool-error Sandbox, git auth, DNS, or MCP tool failed The model burns tokens hallucinating workarounds for a broken environment
reward-hack Agent weakened a test, skipped a check, or searched externally Looks like exit 0 unless an independent verifier checks the diff
confident-wrong Clean exit, plausible summary, broken behavior 68–82% of failed benchmark runs—invisible without independent verification

That HTTP 200 refusal detail is worth tattooing on the wall of every platform team. Since November 12, vendor usage policies and mid-session routing have become a live runtime dependency—enforcement triggers tightened, classifiers reroute or refuse with 200 OK, and served-model headers can diverge from the alias you requested. If your pipeline only checks HTTP status codes, it will happily log a refusal as a completed step.


Five Controls That Actually Move the Number

None of these require waiting for the next model drop. Every one of them belongs in the infrastructure wrapping the agent—not in a polite sentence inside SYSTEM.md.

1. Measure cost per merged change, not cost per million tokens

Attach one trace ID to every pipeline execution and attach child spans to every model call, tool call, and retry carrying: requested_model, served_model, prompt_hash, input_tokens, output_tokens, cached_tokens, and terminal state. Roll it up to dollars per verified, merged diff. If a model is 40% cheaper per token but takes three retries to pass tests, it is more expensive.

2. Enforce retry budgets that require a delta

Hard-cap attempts per step and total spend per execution inside your egress proxy, never inside the agent's prompt. Add one blunt rule: a retry is forbidden unless something changed—trimmed context, a narrowed file scope, a different model tier, or a concrete test failure trace injected cleanly. Repeating an identical prompt after a failure is not resilience; it is a donation.

3. Put prompt-prefix stability under automated test

Because cache reads represent 96–99% of agent token volume, silently invalidating the prefix cache is the fastest way to 4× your bill overnight. Common culprits: injecting Date.now() into the system prompt, non-deterministic tool-schema ordering across turns, or dynamically rewriting instructions mid-loop. Write a unit test that hashes your serialized system + tool prefix across turns 1, 2, and 3 and fails CI if the prefix mutates unexpectedly.

4. Log served_model and assert on stop_reason

Record the actual model returned in response metadata next to the alias you requested, and treat stop_reason: "refusal" or unexpected model downgrades as a first-class degraded pipeline state. Decide in config whether degraded escalates to a human, fails closed, or routes to a fallback provider.

5. Keep the exit path warm with a weekly eval

Vercel's telemetry shows 48.0% of teams switched flagship models in a single month (28.6% switched labs). Switching the API client takes ten lines of code; discovering your prompts and tool schemas silently broke takes three weeks. Run a small, fixed 20-task golden eval through your secondary provider once a week so your fallback is a tested config diff rather than an emergency rewrite.


The Uncomfortable Arithmetic Behind the IPO

It is easy to blame frontier labs for confusing pricing or tighter limits, so let's steelman their position honestly. Building frontier models is extraordinarily capital-intensive. Alex Oppenheimer's breakdown of Anthropic's filing shows that while pure inference gross margin looks like 85.5%, folding in training compute, cloud-marketplace fees, and free-tier/internal usage brings adjusted gross margin down to roughly 28%, with nine-month owner free cash flow around –$4.1 billion. Meanwhile, Q2-to-Q3 price cuts shaved $600 million off quarterly revenue even as $70.7 billion in compute bills come due within a year.

That is not malice. It is fixed-cost arithmetic.

When a supplier has $900.9 billion in long-term compute obligations and public-market shareholders watching operating margin every 90 days, the marginal token is their business model—and your unmonitored retry loop is the easiest revenue they will ever book. Even if list prices never rise, pricing power quietly moves into cache-read rates, tier caps, and fast-mode surcharges.

There is an honest cost to taking ownership of these controls yourself: running a central model proxy, maintaining a versioned eval suite, and enforcing per-run budgets means owning another piece of operational infrastructure—and yes, if your proxy falls over, every agent stops. That tradeoff is real. Small teams shipping a prototype should not build a gateway on day one.

But the moment agents run unattended across dozens of repositories, you cannot negotiate a price—or debug a bill—you cannot attribute to a run.

How is your team tracking cost per merged PR right now, and what was the nastiest silent retry loop you have caught in the wild?

來源:DEV Community · dev.to