3 個降低 LLM 生產環境成本的策略
3 Production Strategies to Dramatically Reduce Your LLM Costs
作者提出降低生產環境 LLM 成本的方法,建議按功能追蹤 token 用量及費用、快取靜態提示詞內容,並按任務複雜度將請求路由至不同模型。文章指出,提示詞快取通常可節省 40-80%,動態模型路由常可減少 40-70% 成本。文中亦建議壓縮提示詞,並以批次 API 處理非即時工作,通常可享 token 費率 50% 折扣。
When deploying AI agents and LLM-powered features to production, API bills can balloon surprisingly fast. Most teams overpay for LLM calls not because the models are inefficient, but because they treat every incoming request the same.
Whether a user is asking a simple intent classification question or requesting complex multi-step reasoning, every request often gets sent to the exact same expensive flagship model.
If you want to run LLMs sustainably in production without sacrificing output quality, you need a structured approach to optimization. Here is how to systematically reduce your LLM infrastructure costs step by step.
Step 1: Instrument Cost Tracking Per Feature
You cannot optimize what you do not track. Before changing your prompts or swapping models, instrument loggers to track input/output token counts and associated costs per endpoint or feature.
Here is a lightweight Python wrapper using standard OpenAI SDK semantics to track usage per feature tag:
import logging
from openai import OpenAI
client = OpenAI()
logger = logging.getLogger("llm_cost_tracker")
# Example pricing mapping per 1M tokens
MODEL_PRICING = {
"gpt-4o": {"input": 2.50 / 1e6, "output": 10.00 / 1e6},
"gpt-4o-mini": {"input": 0.15 / 1e6, "output": 0.60 / 1e6},
}
def call_llm_with_tracking(model: str, messages: list, feature_id: str):
response = client.chat.completions.create(model=model, messages=messages)
usage = response.usage
input_cost = usage.prompt_tokens * MODEL_PRICING[model]["input"]
output_cost = usage.completion_tokens * MODEL_PRICING[model]["output"]
total_cost = input_cost + output_cost
logger.info(
f"Feature: {feature_id} | Model: {model} | "
f"Tokens: {usage.total_tokens} | Cost: ${total_cost:.6f}"
)
return response.choices[0].message.content
Logging costs at the feature level quickly identifies which specific parts of your application burn through your API budget.
Step 2: Implement Prompt Caching
In agentic workflows and Retrieval-Augmented Generation (RAG) systems, system prompts and retrieved context documents are often large and static. Recomputing these identical tokens on every user query is wasteful.
Modern LLM providers support prompt caching. Cached reads cost a fraction of fresh input tokens, often saving 40-80% on long system prompts or static document contexts.
To leverage prompt caching effectively:
- Structure your prompts so static content (system instructions, background context, reference documentation) comes first.
- Keep dynamic content (user query, conversation history) at the very end of the prompt array.
Step 3: Implement Dynamic Model Routing
Not every prompt needs a flagship model. Simple tasks like intent classification, basic JSON extraction, or entity detection can easily be handled by smaller, cheaper models.
Model routing is usually the single biggest lever, often cutting costs by 40-70%. You route routine queries to a lightweight model and reserve the expensive model for hard reasoning tasks.
Here is a production routing pattern using a lightweight classifier before invoking the primary task:
def route_and_execute(user_query: str, context: str):
# Step 1: Determine query complexity using a cheap model
complexity_prompt = [
{"role": "system", "content": "Classify query complexity as 'SIMPLE' or 'COMPLEX'."},
{"role": "user", "content": f"Query: {user_query}"}
]
complexity = call_llm_with_tracking(
model="gpt-4o-mini",
messages=complexity_prompt,
feature_id="routing_classifier"
).strip().upper()
# Step 2: Route to the appropriate model based on classification
selected_model = "gpt-4o" if "COMPLEX" in complexity else "gpt-4o-mini"
task_prompt = [
{"role": "system", "content": "You are a helpful technical assistant."},
{"role": "user", "content": f"Context: {context}\n\nQuestion: {user_query}"}
]
return call_llm_with_tracking(
model=selected_model,
messages=task_prompt,
feature_id="core_task"
)
By adding a fast, low-cost classifier step, a large portion of routine traffic can bypass the flagship model entirely.
Step 4: Prompt Compression and Batching
Once tracking, caching, and routing are live, refine payload sizes and execution timing:
- Prompt Compression: Trim verbose system instructions, strip unnecessary historical messages from conversation memory, and remove redundant formatting tokens.
- Batch Processing: For non-realtime background workloads (such as nightly document indexing or batch evaluation), use provider batch APIs, which typically offer a 50% discount on token rates.
By applying this structured sequence—instrumenting cost tracking first, followed by caching, routing, and compression—you can optimize production LLM workloads effectively without dropping output quality.
About me
Tanmay builds AI agents and automation for businesses and is open to freelance projects. Feel free to connect with me on LinkedIn.
來源:DEV Community · dev.to