我如何把每個 AI 智能體任務路由至仍有配額的最低價模型,以及過程中的問題
I Route Every AI Agent to the Cheapest Model with Quota. Here's What Broke
作者用 Python 編寫路由器,將每項 AI 智能體任務交給仍有配額的最低價模型。路由器會略過付費模型及配額窗口使用率超過 90% 的模型;模型在啟動首 60 秒內失敗時,會改試同一級別的下一個候選模型,但不會自行升級級別。低價模型曾寫出令測試無限循環、記憶體升至 9 GB 的測試,也曾在腳本加入 419 行重複輔助函式。
I wanted coding with AI agents to be genuinely cheap, so I built a tiny Python router that sends every task to the cheapest model that still has quota left.
Within three days, a cheap model wrote a test that looped forever and ate 9 GB of RAM, another stopped halfway with "Free usage limit reached", and a third dumped 419 lines of duplicated helper functions into a script because it assumed "the grader supplies tests." 😅
Everyone is asking whether coding with AI is actually faster and cheaper. My honest answer? It's fast and nearly free, right up until a cheap model writes something that takes two hours to unwind.
So here's the router, the rules that came out of those crashes, and why it's forbidden from upgrading itself.
TL;DR:
- Models live in three tiers (small, medium, large), each sorted cheapest first.
- The router skips paid models and anything above 90% of its quota window.
- If a model dies in its first 60 seconds, it tries the next one in the same tier.
- It never escalates to a bigger tier on its own. A human decides that.
Agents have roles; tiers have price tags
The first mistake I made was tying each agent prompt to a model.
Now the role ("writer", "designer", "coder") is just a prompt with rules and skills, and a role never names a model. It only says what size of work it usually does.
Why does that matter? Because the moment a model is hardcoded, you can't route around outages, renamed models or an exhausted rate limit. The tier list does that for you:
{
"small": ["free-model-a", "free-model-b", "cheap-model-c"],
"medium": ["mid-model-a", "mid-model-b"],
"large": ["big-model-a", "big-model-b"]
}
Cheapest first, more capable after. Swapping a model is a one-line change, and no prompt has to know.
The 20-line candidate filter
When a task comes in, the router walks the requested tier and filters it:
def select_candidates(tier, tiers, allow_paid=False, forced=None):
if forced:
return [forced], []
to_try, skipped = [], []
for cand in tiers[tier]:
if cand.get("paid") and not allow_paid:
skipped.append((cand, "paid model not allowed"))
continue
if (q := quota_status(cand)):
skipped.append((cand, f"quota: {q}"))
continue
to_try.append(cand)
return to_try, skipped
quota_status() reads the CLI's local usage numbers instead of hitting a rate-limit API. If any window (rolling hours or weekly) is above 90%, that model is out.
Why 90 and not 100? Start a big task at 96% and it tends to run dry mid-stream, leaving half-written files behind. The buffer is cheaper than the cleanup.
And yes, you can force a model by hand. The router just warns you and gets out of the way.
The 60-second escape hatch
What about a model that gets picked and then falls over before doing anything?
AI CLIs fail at startup in creative ways: renamed models, instant 429s, free tiers that cut you off at second zero. So the router watches the first minute:
STARTUP_ERRORS = ("429", "model_not_found", "Free usage limit", "quota")
def is_startup_failure(code, seconds, log):
if seconds > 60:
return False
if any(e.lower() in log.lower() for e in STARTUP_ERRORS):
return True
return code != 0
Fail fast, and it rolls to the next candidate in the same tier. Every run also leaves its prompt, raw log, final answer and the choice it made, so I can see why a model was skipped without guessing.
What broke when I trusted the cheap end
Did the router make everything smooth? Not even close. A few of my favourite disasters:
-
The phantom model. A model showed up in the CLI's own model list and failed instantly with
model_not_found. Lesson: only route to models you've actually run. - The quota wall. A free model stopped halfway with "Free usage limit reached", and another refused to run because of where it was hosted.
- 419 lines of spaghetti. A free small-tier model was asked for a tiny flag in a script. It added ~20 duplicated helper functions, changed existing signatures and wrote zero tests. Full revert.
- The 9 GB loop. A cheap model wrote a pagination test whose mock always returned a full page. The loop never ended and memory climbed to 9 GB before the process got killed. Now every generated test runs under a memory cap and a timeout.
- The quota burn. One long high-effort run ate about 30% of a weekly allowance in a single go.
(If you like this kind of thing, my previous post on running a side project with AI agents has more.)
Why the router never escalates
When every small model fails, why not bump the task to a bigger one automatically?
Because the router never decides that a task is bigger. Only I do.
When a small task fails across the board, it's rarely the model being dumb. It's usually:
- a vague prompt,
- a task touching too many files,
- or a broken test setup.
Auto-escalating just pays premium prices to retry a broken prompt. So when a tier runs out, the router exits loudly and lists why it skipped each model. Then I either fix the prompt or bump the size by hand.
The machine executes; the human owns the scope.
So did it actually save anything?
I don't have a dollar figure, since most of this runs on subscriptions and free tiers rather than metered APIs.
What it saved was headroom and attention:
- Drafts, copy and small single-file edits run on free or cheap models.
- No more surprise rate-limit lockouts in the middle of the day.
- Startup failures fall back to another model before I even notice.
The one rule I'd keep if I had to delete everything else:
Cheapest-with-quota is the right default for writing and small scoped edits. For anything with tests, set a higher floor.
Saving ten minutes of quota is worthless if you spend an hour reverting a cheap model's "tests".
Over to you 👀
I use this router to keep a few small data sites running on my own, like a cheapest fuel price map for Spain and a public tenders finder.
The agents are the easy part. Getting people to find and use the sites is the hard one. So, for anyone who's grown a side project:
- What brought you your first steady search traffic on a site with thousands of auto-generated pages?
- Ads, paid alerts, API or datasets: what actually paid for a niche data site?
- Where would you share a site like this besides dev.to?
- What would you stop doing if you were me?
I read and reply to every comment 💛
I write these with help from AI agents.
來源:DEV Community · dev.to