作者分享如何將 AI 推理成本削減 80%，由 OpenAI 遷移至自託管模型
How I cut my AI inference bill by 80% - moving from OpenAI to self-hosted models
作者分享將 AI 推理由 OpenAI 遷移至 Hetzner 單台 GPU 伺服器自託管，稱月費由 $400 降至 $80。文中的成本表則列出總成本由每月 $490 降至 $195，節省約 $295；配置為 1× A100 80GB，透過 vLLM 執行 Qwen 2.5 72B，並使用 LiteLLM。
My OpenAI bill was $400/month. After moving to self-hosted models on a single Hetzner GPU box, it's $80/month and falling. Here's the playbook.
The OpenAI bill
I was using GPT-4o for ~3M input tokens/day for a RAG app and ~500K output tokens/day. About $450/month. The killer was the output tokens.
What I tried
Option 1: Smaller OpenAI models (gpt-4o-mini)
Worked for ~30% of queries. Quality drop too steep for the rest. Saved ~$100/month.
Option 2: Llama 3.1 70B on RunPod
Cost: $1.50/hour for an A100, mostly idle. About $300/month. Same quality as GPT-4o on most of my tasks. Breakthrough.
Option 3: Self-hosted on Hetzner
Hetzner GPU servers (H100, A100) in Frankfurt. Latency ~10ms from EU, ~95ms from US. Worked for me.
What I actually run
Hardware: Hetzner GPU server, 1× A100 80GB on-demand. Effective cost ~$190/month (shared between workloads).
Software: vLLM serving Qwen 2.5 72B (Apache 2.0), all-MiniLM-L6-v2 for embeddings, LiteLLM as drop-in OpenAI-compatible gateway.
Cost breakdown
| OpenAI | Self-hosted | |
|---|---|---|
| LLM inference | $400/mo | $190/mo |
| Embeddings | $60/mo | $5/mo |
| Vector DB | $30/mo | $0 |
| Total | $490/mo | $195/mo |
Savings: ~$295/month = $3,540/year.
The catch
You need:
- Linux + Docker + Python comfort
- Willingness to debug CUDA driver issues
- Capacity to handle 4-second cold-start latency
If you have $500+/month OpenAI spend, the math works.
What I gave up
- Peak latency: OpenAI p99 = 800ms. Mine = 2.5s.
- Model variety: Locked to one model. OpenAI gives easy access to GPT-4o, o1, o3-mini.
- API stability: vLLM crashes occasionally. OpenAI never.
- 24/7 on-call: When vLLM goes down at 3am, I get paged.
What surprised me
- Qwen 2.5 72B is competitive with GPT-4o on most coding/writing tasks.
- vLLM throughput is amazing — 50+ requests/sec on a single A100.
- Embeddings are dirt cheap to self-host — all-MiniLM-L6-v2 is 80 MB, runs anywhere.
- Hardware isn't the bottleneck. Model loading + tokenization is.
My migration playbook
- Measure current bill. Know what's hurting.
- Pick highest-volume task. Start there.
- Stand up vLLM locally. Test with your prompts.
- Add LiteLLM in front. Drop-in OpenAI replacement.
- Move 10% of traffic. Compare outputs.
- Move 100% if quality OK. Save the bill.
- Set up monitoring. vLLM doesn't alert on crash.
When NOT to self-host
- Under $200/month on OpenAI
- Need o1-class reasoning
- No GPU/LLM ops experience
- Can't tolerate 24/7 on-call
Cost projection
If I grow to 2x volume:
- OpenAI: ~$980/month
- Self-hosted: needs 2× A100. ~$700/month
Savings drop as you scale. But at current scale, savings are real.
Final thought
AI inference is now genuinely cheap to self-host if you have the volume to amortize setup. If you're paying $400+/month to OpenAI, it's worth investigating.
來源：DEV Community · dev.to