Elias Biondo 以 Rust 開發 Antirion，稱其為全球最快 AI 閘道

TL;DR

I built Antirion, an open-source (Apache-2.0) AI gateway written in Rust: one binary between your apps and LLM providers, with routing, fallbacks, budgets, guardrails, a request log and a built-in web console.

I benchmarked it against Bifrost, the open-source gateway written in Go, on the same 4 vCPU machine, with simple chat at 5,000 req/s:

LiteLLM ran in the same benchmark and failed every request at that rate. The rest of this post covers the design decisions behind those numbers, and how I measured them without fooling myself.

Why an AI gateway, and why speed matters

If you've shipped LLM features to production, you know how it goes. It starts with one OpenAI key in a .env. Then someone wants to try Claude. OpenAI has an outage on a Friday afternoon and there's no fallback. Finance asks what each team is spending. Legal asks whether customer data is ending up in prompts.

The usual answer is an AI gateway: a proxy between your apps and the providers. Your apps keep speaking the OpenAI (or Anthropic, Gemini, Bedrock) API, and the gateway handles routing, fallbacks, rate limits, budgets, guardrails and logging.

The catch is that a gateway sits on the path of every single call. If it adds 20 ms, that's 20 ms on every request your company makes, forever, and 20 ms is about what Bifrost added at 5,000 req/s in my benchmark. The ones I tried got slow under load, used more memory than a proxy should, and needed several services just to get a dashboard. So performance was a requirement from the first commit.

Here's what using it looks like. Your code doesn't change, only the base URL:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-antirion-...")

# "chat" is a Model defined in the gateway. It can point at GPT, Claude,
# Gemini or a local vLLM server, with fallbacks behind it.
response = client.chat.completions.create(
    model="chat",
    messages=[{"role": "user", "content": "Hello"}],
)

Now, the decisions that made it fast.

1. One thread per CPU instead of work-stealing

Tokio's default is a multi-threaded runtime with work-stealing: any thread can pick up any task. That's great for uneven workloads, but it has a cost. Tasks migrate between CPUs, caches go cold, and waking a task on another thread means cross-thread synchronization.

Antirion runs one data thread per CPU instead, each with its own single-threaded runtime:

• Accept: on Linux, each thread listens on its own socket in a SO_REUSEPORT group, and the kernel spreads connections across them. There's no central accept loop handing connections around.
• Affinity: a client connection, the tasks it spawns and its HTTP/1.1 connections to the provider all stay on the same thread. The outbound connection pools are keyed by thread, so a request never wakes another thread from start to finish.
• A separate admin plane: the console, the Admin API and background jobs run on their own multi-threaded runtime, so they never compete with the hot path.

This model has two weak spots, and both needed handling:

• HTTP/2: one connection can carry hundreds of concurrent requests. If they all stay on the thread that accepted the connection, one heavy client pins a CPU while the others sit idle. So the requests of an HTTP/2 connection are spread across the data threads in turn.
• Outside Linux: without the same SO_REUSEPORT balancing, a single accept loop hands each connection to the data thread serving the fewest.

2. Don't copy the request body

Long-context prompts easily reach hundreds of KB, and the gateway usually only needs to touch one or two fields, like rewriting model to the name the provider expects. Deserializing into a tree, editing it and serializing it again costs CPU and memory in proportion to the prompt.

So I wrote a small JSON parser that does two things in one pass:

• Strict validation: depth limits, duplicate keys, and errors with line and column. Duplicate keys are a classic proxy attack vector, since the proxy may read one value and the upstream another.
• Member spans: it records where each top-level member starts and ends, in bytes.

When the request goes out, the body is a Payload: slices of the client's original bytes around the edited member, each sent as its own frame. When the provider speaks the same dialect as the client, the body is never copied. The pieces are joined only when a provider needs to edit members.

3. Translate between dialects, and never try the impossible

An OpenAI SDK client can call Claude, and the other way around. There's an intermediate representation in the middle: each client dialect (Chat Completions, Responses, Anthropic Messages, Gemini, Bedrock Converse) has a decoder into it, and each provider has an encoder out of it.

The trick that paid off most: encoding runs once as a dry run before routing, and the verdict is cached per deployment. If a deployment can't serve the request (a feature with no equivalent on the target, for example), it's dropped from the candidates before the first attempt. Without this, a fallback would spend a round trip to the provider just to find out it was never going to work.

Streams are translated event by event. The SSE parser keeps each event's original bytes, so when no translation is needed, events pass through without being rewritten.

4. Streams with a commit point and a byte-bounded queue

Streaming is where LLM proxies get tricky. Before the first byte reaches the client, you can still switch providers if the upstream fails or a guardrail blocks. After the first byte, you can't.

So the response is held until a commit point. From there, a dedicated task relays events through a queue bounded in bytes, not in messages. The HTTP body releases queue space as the server takes frames, so a slow client pushes back on the upstream instead of piling up memory in the gateway. When the client disconnects, the queue closes and the relay stops.

5. Immutable configuration, swapped by generation

The whole configuration is one document (YAML, the Admin API or the console), validated against a JSON Schema. Every change becomes a new generation: the compiled snapshot (routes, limits, prices, keys) is built on a blocking thread, persisted, and swapped in at once.

The snapshot lives in an ArcSwap. Each request loads it once at the start and keeps it to the end, so a config change never catches a request halfway, and the hot path never contends for a lock with whoever is writing configuration. State that has to survive generations, like rate-limit counters, lives separately, keyed by stable identities.

6. Charge every request exactly once

Limits and budgets are checked at admission with a reservation based on a token estimate. When the request finishes, it settles exactly once with the real usage, writing one ledger line per upstream call.

One detail that matters: every request runs on a tracked task, not tied to the connection. If the client disconnects halfway, the request isn't cancelled, and the cost of what the provider already generated is still accounted for. If a task panics with a reservation open, the reservation is refunded.

7. A cache with a canonical key and single-flight

The exact cache keys on a canonical form of the body (JSON canonicalized per RFC 8785), so key order or whitespace never causes a miss. When several identical requests miss at the same time, only one goes to the provider and the others wait for its answer.

8. An architecture the build won't let you break

The code is 28 crates in 8 layers, from the JSON parser up to the binary, and a crate may only depend on layers below it. A cargo xtask layers check runs in CI and fails the build if anyone breaks the rule. The allocator is jemalloc on Linux with glibc and mimalloc elsewhere.

9. Measuring without fooling yourself

This part taught me the most. The most common mistake in proxy benchmarks is closed-loop load: the generator waits for each response before sending the next one, so when the proxy stalls, the generator slows down with it and the queue never shows up in the latency. That's coordinated omission.

The harness I wrote follows these rules:

• Open loop: every request leaves at its scheduled time, and latency counts from that time, not from when it actually left.
• Separate machines: the load generator and the mock upstream run on c7i.4xlarge instances, the gateway on a c7i.xlarge (4 vCPUs), all in one cluster placement group. The mock answers at once from pre-rendered bodies.
• Overhead by subtraction: each repetition runs twice at the same rate, once through the gateway and once straight to the mock. The overhead is the difference between the two at the same percentile.
• Interleaved: in every repetition the three gateways run in turn, so any drift of the machines hits all of them alike. Each process starts fresh for every repetition, and the result is the median of five.
• Validity rules: a repetition is thrown out if more than 0.1% of requests leave over 1 ms late, if the generator or the mock goes above 60% CPU, or if the direct run isn't clean.
• Saturation: the highest rate at which a gateway achieves at least 99.5% of the offered rate, with at most 0.01% errors and a p99 overhead of at most 5 ms.

One surprise along the way: the self-test required a 50 µs p99 from the generator to the mock, and it never passed on AWS. A bare TCP connect between two machines in the same placement group already takes 0.5 to 0.8 ms at p99. Since overhead subtracts each repetition's own direct run, this doesn't affect the comparison, but it shows how much the network weighs on a sub-millisecond number.

Antirion vs Bifrost

Every gateway ran the same minimal setup: one key checked on every request, no request logging, no cache. Bifrost 2.2.4 got the same pool settings Bifrost uses in its own published benchmark (initial_pool_size 15,000, buffer_size 20,000, concurrency 10,000), so it wasn't running on defaults that would hold it back.

The headline is in the TL;DR. The other scenarios tell the same story, and the gap grows with load:

At 10,000 req/s Bifrost still answers every request, but they queue: the median request waits 12 seconds. Antirion stays under 2 ms at p99.

To be fair to Bifrost: at a light load the two are close. At 1,000 req/s Bifrost's p50 was 0.72 ms against Antirion's 0.76 ms, a tie within noise. Antirion's p99 was lower (0.95 ms against 1.16 ms), and it used 2.6x less CPU per request. The difference shows up as load grows, which is exactly when a gateway matters.

LiteLLM 1.103.1 ran in the same benchmark. It failed every request at 5,000 req/s and stayed below 500 req/s at saturation.

The full method, the configs of all three gateways and the steps to rerun it on your own AWS account are in docs/benchmarks.md. Benchmarks against other projects are easy to get wrong, so if anything in the setup looks unfair, please tell me.

Try it in two minutes

All you need is Docker and an OpenAI or Anthropic key. No config file:

docker run -p 8080:8080 -p 8081:8081 \
  -v antirion-data:/var/lib/antirion eliasbiondo/antirion

The first start prints the console URL and an admin token. Open http://localhost:8081/ui/, add a provider, create a Model named chat, create an API key, and point any OpenAI SDK at http://localhost:8080/v1. Every request shows up in the request log with its route, tokens, latency and cost.

For production there's Docker Compose, a cluster mode on PostgreSQL and Valkey, and a Helm chart.

What's next

Antirion is at v0.1.1, so there are rough edges, and I'd love your feedback:

• What gateway are you using today, and what bugs you most about it?
• Would you hold a stream until a commit point, or send the first byte as early as possible and give up on fallbacks?
• Anything in the benchmark method you'd challenge?

If you found this useful, a ⭐ on GitHub helps a lot. It's how other people find the project.