SGLang 與 vLLM：Agent 選錯 KV cache 會拖慢速度

If you serve LLM agents in production, you have probably seen this week's numbers: in a multi-turn agent workload with an 8,000-token system context, SGLang's RadixAttention cut time-to-first-token (TTFT) from 440ms to 38ms — an 11.57x speedup — while vLLM v0.6 with prefix caching managed 480ms down to 92ms, a 5.21x improvement (Qwen2.5-Coder-32B-Instruct-AWQ), as benchmarked by NextByteTech in their Dev.to post on ollama vs vLLM vs SGLang in 2026.

The data is real, and the conclusion is mostly right: for long-context, multi-turn agent traffic, SGLang currently wins. But "mostly" hides the interesting part. Those speedups only materialize when your workload has the right shape. Run the same two engines against different traffic and the gap collapses — or flips. This post is the decision framework the benchmark post didn't give you: when RadixAttention genuinely pulls ahead, and when both engines tie.

Why RadixAttention Wins in Agent Workloads

Both vLLM's prefix caching and SGLang's RadixAttention reuse previously computed KV cache for repeated prompt prefixes. The difference is the data structure.

vLLM's automatic prefix caching (APC) historically hashes whole blocks of the prompt and matches them against cached blocks. It's effective when requests share long, identical prefixes — the classic system-prompt-plus-few-shot-template shape — but it degrades when prefixes diverge mid-sequence or when the overlap doesn't align to block boundaries.

SGLang's RadixAttention stores the KV cache in a radix tree (a compressed trie) over token sequences. Repeated prefixes become shared tree paths, and the cache can be reused at the granularity of any common prefix, not just aligned blocks. Multi-turn agent conversations are exactly this: the same 8,000-token system context plus growing conversation history, turn after turn. Each new turn shares a massive common prefix with the previous one. The tree reuses nearly all of it, which is why the reported TTFT dropped to 38ms — almost the entire prefill was skipped.

The second Dev.to comparison (ifnodoraemon's SGLang vs vLLM architecture showdown) makes the mechanism visible in the metrics: at a 4,096-token prompt with 75% prefix overlap and concurrency 64, SGLang hit a 78.6% KV cache hit rate vs vLLM's 41.2%, with median TTFT of 85ms vs 380ms (4.47x), P99 TTFT of 280ms vs 1,250ms, and throughput of 5,430 vs 3,120 tok/s. The hit-rate gap is the performance gap. When your workload is prefix-overlap-heavy, the radix tree's finer-grained reuse shows up directly in every latency percentile.

But Sometimes They Tie

Here is where the story gets useful. Three published counter-cases show where the advantage evaporates.

1. Unique-prompt traffic: the gap nearly disappears

bex.co's September 2026 report (SGLang Crossed 400,000 Production GPUs) benchmarked Llama 3.3 70B FP8 on Spheron H100s at 50 concurrent requests with unique prompts — traffic with no shared prefix to reuse. The result: SGLang at 1,920 tok/s vs vLLM at 1,850 tok/s, with TTFT within 20ms of each other. No shared prefix, no cache to share; the radix tree has nothing to work with, and the two engines are functionally tied.

2. Raw throughput on short prompts: vLLM can win

The same bex.co report includes a May 2026 result on a single H100 with Qwen2.5-7B: vLLM delivered 23,500 tok/s of raw throughput vs SGLang's 16,800 tok/s — a clear vLLM win on aggregate tokens per second. SGLang still held a 4x+ TTFT advantage (1.8s vs 7.8s), so it wasn't a loss on every axis, but for a throughput-first, latency-tolerant batch workload, the "SGLang is faster" headline would have been wrong.

3. High concurrency: the crossover is real

bex.co's DGX Spark concurrency sweep found SGLang winning at 1–4 concurrent requests, with vLLM winning at 6+ concurrent requests. And RunPod's blog on the SGLang vs vLLM KV cache comparison (2x H100, DeepSeek-R1-Distill-Llama-70B, 7k context) concluded that a cache hit yields roughly +20% speed — and that on fresh context the two engines are comparable. If your traffic is many short, independent prompts at high concurrency — the classic chatbot-serving shape — the radix tree overhead buys you nothing and vLLM's throughput engine takes the round.

The Decision Checklist

Before you switch engines, measure your workload against these four questions:

1. How much prefix overlap do you actually have? If most of your requests share a long system prompt or conversation history (agents, RAG with fixed instruction blocks, few-shot templates), RadixAttention's finer-grained reuse pays off. If every prompt is unique, the tree is dead weight.
2. What's your concurrency? Published sweeps show a crossover: SGLang ahead at low concurrency, vLLM ahead as concurrency climbs (around 6+ on DGX Spark in one report — verify against your own hardware). Size for your actual traffic, not the benchmark's.
3. Are you prefill-heavy or decode-heavy? Prefix caching only accelerates the prefill phase. If your outputs are long and your prompts are short, most of your time is in decode, and the cache advantage shrinks to a rounding error. RunPod's +20%-on-cache-hit figure is a good mental anchor.
4. Which latency percentile matters to you? SGLang's biggest reported edge is in TTFT and tail latency (P99 280ms vs 1,250ms in the 75%-overlap scenario). If your SLO is on end-to-end tokens-per-second for batch jobs, that edge may not buy you anything — vLLM's raw throughput win shows the other side of the trade.

Conclusion

The benchmark post's headline number — 38ms vs 92ms TTFT in the multi-turn agent scenario — is correct for that workload, and RadixAttention's radix-tree design is genuinely the better mechanism for shared-prefix traffic. But engines don't run benchmarks; they run your traffic. Unique-prompt serving, short-prompt batch throughput, and high-concurrency chatbot shapes all show the two engines tying or vLLM winning, per bex.co, RunPod, and the second Dev.to comparison. Measure your prefix overlap and concurrency first, then pick the cache that fits.

We write about production LLM infra and agent systems for builders shipping real products. Follow along here:
https://t.me/qisuancloud
https://qisuanai.com