跳到正文
DEV Community· Ahmed Jaffar·· 14 天前AI 評分61

AI 推理的真實成本不在 token 價格,而在可接受答案的成本

The real cost of AI inference is not the token price

AI 導讀

Ahmed Jaffar 指出,AI 推理成本應按每個通過交付前檢查、可實際採用的答案計算,而非只看每百萬 token 的價格。他將重試流量和成功率納入成本,並主張先定義可接受輸出,再比較不同模型配置,按錯誤代價和請求量分配流量。

正文

Ahmed Jaffar

Most teams track their AI spend the wrong way. They look at the price per million tokens and call that the cost. It is not. The cost that matters is what you pay for an answer you can actually use.

I learned this building a small toolkit I call the AI Inference Control Plane. It runs model configurations against real tasks and measures cost per accepted output. An output counts as accepted only if it passes the checks you would apply before shipping it.

The gap between the two numbers is where the money goes. A cheaper model that needs three retries costs more than an expensive model that gets it right once. A fast model that produces answers your team throws away is not fast at anything that matters.

Three things changed how I think about inference cost:

  1. Retries are a cost line. If your failure rate is 30 percent, your real price is not the sticker price. It is the sticker price divided by your success rate, plus the retry traffic on top.

  2. Evaluation has to come before optimization. Until you can say which outputs are acceptable, every cost comparison is a guess. Teams that struggle are usually optimizing a number they never defined.

  3. Configuration is a portfolio. The right setup is rarely one model for everything. It is a mix: a strong model where errors are expensive, a cheap model where volume is high and mistakes are cheap, and clear rules for which traffic goes where.

None of this needs new tooling to start. Pick your ten most common requests. Run two or three configurations against them. Count what you paid for the answers you kept. That ratio, cost per accepted answer, is the number to manage.

The AI bill is not the problem. The undefined cost is. Once you measure the right unit, the decisions get much easier.

The toolkit is open source if you want to look at how it measures this: github.com/AhmedJaffar/AI-Inference-Control-Plane

Originally published on my Substack: https://ahmedjaffar.substack.com/p/the-real-cost-of-ai-inference-is

來源:DEV Community · dev.to