SanVi 開源 Kubernetes AI 閘道,支援預算控制、緊急停用及 CI 評測
I built an open-source AI gateway for Kubernetes: budgets, kill switch and evals in CI
作者開發並開源 SanVi,一個面向 Kubernetes 的 AI 閘道,讓應用程式透過 OpenAI 相容端點及各自的虛擬金鑰存取模型,並為每個金鑰設定模型、預算和速率限制。
Every team wants AI. Then three questions arrive: What does it cost? Where are the keys? Can we stop it?
I'm a platform engineer, and I built SanVi to answer all three on Kubernetes.
The gateway
Apps call one OpenAI-compatible endpoint (LiteLLM engine) with their own virtual key. Each key has its own models, budget and rate limit. No key → 401. Over the rate limit → 429. Over budget → blocked. Every call is logged with team, model, tokens and cost in Postgres.
Keys never in code
The provider key lives only in Vault and syncs in through External Secrets. I rotated it: about 70 seconds, no redeploy.
The kill switch
Leaked key, runaway agent loop, spend spike: kill_switch.sh block <team> returns 401 in seconds, and unblock restores it. Nothing is deleted.
AI access as code
Teams, models and budgets live in access/teams.yaml. Changes go by pull request; CI checks policy (a greedy $50 budget was blocked); Argo CD applies on merge.
Evals decide, not opinions
The Right-Size Advisor runs the same eval suite on each model. On my workload Haiku matched Sonnet: 95% cheaper, 40% faster. Any change that drops below 80% fails CI.
Human in the loop
The deployment agent turns one sentence into a validated Dockerfile and Helm chart, but it can only open a pull request.
Audit-ready
A compliance script checks 12 controls (no secrets in Git, least-privilege access, audit trail, kill switch, quality gate...) mapped to ISO 27001, SOC 2 and EU AI Act themes, on every change.
Try it
Repo, Quickstart and Helm chart: https://github.com/sandhilk/sanvi-gateway
It's a personal project, so blunt feedback is welcome. What would you need before running this in your company?


來源:DEV Community · dev.to