模型路由如何降低推理成本
Using model routing to lower inference costs
模型路由會按任務複雜度、推理需求等條件自動選擇模型,讓不同請求在品質、延遲與推理成本之間取得平衡。文章指出,純按成本計算,日活用戶超過 100,000 時模型路由可能已能抵銷成本,推理成本降幅常見為 50%,95% 亦非罕見。但若只把最複雜的 20% 查詢交給前沿模型,中等複雜度案例可能被分配至無法正確作答的低成本模型,形成 Pareto trap,令節省轉化為客戶留存及支援成本。
Contributing Writer
feature
Oct 21, 20266 mins
Running frontier models for coding and other purposes can be very expensive, but also, sometimes, very effective. It’s tempting to use smaller or older models instead of frontier models because they’re cheaper, although they may not work as well. The ideal model for any given task tends to depend on the complexity of the task. Picking the most appropriate model automatically based on the request and the context is called model routing.
How model routing works
There are several ways of implementing model routing. In general, model routing depends on the pool of available models, the criteria used for selecting a model, and the objective that the router is trying to optimize. Different vendors offer different implementations of this scheme.
Databricks’ Unity Gateway has (in beta) Smart Routing for Codex or Claude Code. Smart Routing automatically selects the lowest-cost model capable of handling each task, so developers do not need to choose a model for every request.
Microsoft’s model router, a trained language model, analyzes your prompts in real time based on complexity, reasoning, task type, and other attributes. It does not store your prompts. It routes only to eligible models based on your access and deployment types, honoring data zone boundaries. You can choose balanced mode, cost mode, or quality mode.
Not Diamond Code uses a lightweight local proxy that computes the derived metadata needed for routing to evaluate the broader session, including expected downstream work and cache state, and select the model and reasoning effort that optimize expected quality and total session cost. It uses more capable models when they are likely to improve the outcome or finish in fewer steps, and more cost-efficient models when additional intelligence is unlikely to add value.
NVIDIA’s NeMo Switchyard has an array of routing algorithms that it can use. These include the LLM classifier, which uses an LLM as a judge to select a candidate LLM and maintains session affinity with that model across later turns; the stage router, which examines recent tool activity to decide how much model capability the agent needs; the escalation router, which starts each conversation with a lower-cost model and moves the session to a more capable model when it detects sustained difficulty; and the trainable prefill router, which learns from workload data to predict model success likelihood.
OpenRouter’s Auto Router analyzes your prompt and selects a model based on what the OpenRouter community, in aggregate, uses for that kind of work. The selection is constrained by your cost tier and guardrails. If the automatic primary pick fails, the router uses a fallback.
Replit’s Intelligent Model Routing matches tasks to the model they’re best suited for, aiming to balance quality, speed, and cost.
Snowflake’s Cortex AI Gateway uses dynamic model routing to select the most affordable model that can confidently complete the task at each step of agent execution. Lower-complexity and repetitive tasks can be directed to more efficient models, while workloads that require deeper reasoning can be routed to frontier models.
Also worth noting: the RouteLLM (2024) paper proposed several efficient router models that dynamically select between a stronger and a weaker LLM during inference, aiming to optimize the balance between cost and response quality. It also proposed a training framework for these routers that takes advantage of human preference data and data augmentation techniques to enhance performance.
The costs and benefits of model routing
Model routing potentially provides massive cost reductions, lower average latency, optimized quality, reduced vendor lock-in, enhanced privacy, and improved operational resilience. Strictly on a cost basis, model routing is likely to pay for itself above 100,000 daily active users. Even if you only save 20% on your LLM bill, the savings can be significant, but 50% cost reduction is common, and a 95% cost reduction is not unheard of.
Inference costs, alas, don’t tell the whole story. Some indirect costs include quality evaluation overhead, human review, and infrastructure.
There’s also a nasty failure mode that can, for example, turn your inference cost savings into customer retention and support costs that outweigh them by a factor of four to five: the Pareto trap. To understand that, we should start with Vilfredo Pareto’s 1906 observation that 80% of the land in the Kingdom of Italy was owned by 20% of the population. That was adapted in 1941 by Joseph M. Juran to apply to quality control: 80% of problems stem from 20% of the causes. The Pareto trap refers to concentrating on and over-optimizing for the top 20% to the exclusion of the remaining 80%.
Essentially, tuning your model router to handle only the top 20% of queries (in complexity) with a frontier model risks sending too many of the edge cases (the moderately complex queries) to the cheaper model that is unable to answer them correctly. That can lead to hidden failures by the cheap models which in turn cause expensive impacts on the business.
There are multiple steps you can take to avoid this trap. One is broader monitoring, enough to bring the failures to light even if they show up in a different department. Another is more conservative tuning of the model router. And third, you can use a more sophisticated routing model to predict query complexity more accurately.
Model routing optimizes the selection of appropriate models for specific tasks, balancing cost, quality, and latency. Various vendors implement model routing differently, utilizing factors like task complexity, reasoning, and cost to determine the best model. While model routing offers significant cost savings and improved performance, it also poses challenges like the Pareto trap, where over-optimization for cost can lead to hidden failures in simpler queries.
Contributing Writer
Martin Heller is a contributing writer at InfoWorld. Formerly a web and Windows programming consultant, he developed databases, software, and websites from his office in Andover, Massachusetts, from 1986 to 2010. From 2010 to August of 2012, Martin was vice president of technology and education at Alpha Software. From March 2013 to January 2014, he was chairman of Tubifi, maker of a cloud-based video editor, having previously served as CEO.
Martin is the author or co-author of nearly a dozen PC software packages and half a dozen Web applications. He is also the author of several books on Windows programming. As a consultant, Martin has worked with companies of all sizes to design, develop, improve, and/or debug Windows, web, and database applications, and has performed strategic business consulting for high-tech corporations ranging from tiny to Fortune 100 and from local to multinational.
Martin’s specialties include programming languages C++, Python, C#, JavaScript, and SQL, and databases PostgreSQL, MySQL, Microsoft SQL Server, Oracle Database, Google Cloud Spanner, CockroachDB, MongoDB, Cassandra, and Couchbase. He writes about software development, data management, analytics, AI, and machine learning, contributing technology analyses, explainers, how-to articles, and hands-on reviews of software development tools, data platforms, AI models, machine learning libraries, and much more.
More from this author
來源:InfoWorld · infoworld.com