vLLM KV Cache 計算解析：24GB GPU 為何同時只能處理 4 個請求

I put Llama 3.1 8B behind vLLM on a single 24GB card, pointed a load test with 20 concurrent users at it, and watched the stats line:

Running: 4 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.1%

Four. I had been told vLLM does continuous batching and can run hundreds of sequences at once. The default max_num_seqs is 256. GPU compute utilization was nowhere near maxed. Sixteen users were sitting in a queue anyway.

The answer had been in the startup log the whole time, and it comes down to vLLM KV cache math you can do on a napkin. Once you do it, you stop guessing how many users a GPU can serve.

TL;DR

• vLLM's concurrency ceiling is set by KV cache memory, not compute and not max_num_seqs.
• KV cache cost per token = 2 × layers × kv_heads × head_dim × bytes. For Llama 3.1 8B in bf16 that's 128 KiB per token.
• On a 24GB GPU, bf16 weights eat ~15 GiB, leaving roughly 5 GiB for KV cache: about 40,000 tokens total, shared by every request in flight.
• At ~8,500 tokens per RAG request, that's 4 concurrent requests. The rest wait, or get preempted mid-generation.
• Biggest levers: --kv-cache-dtype fp8 (halves per-token cost), quantized weights (frees ~9 GiB), prefix caching for shared prompts. Lowering --max-model-len does not grow the pool.

What is the KV cache in vLLM, and why does it eat memory?

The KV cache stores the key and value vectors for every token a sequence has seen, in every layer, so the model doesn't recompute attention over the whole prompt each time it generates a new token. It grows linearly with sequence length, and every active request needs its own copy (unless prefixes are shared).

That's the trade. Without the cache, each new token would mean re-running the full prompt through all 32 layers. With it, you pay in memory instead of compute. On a small GPU, memory is the thing you run out of first.

How do you calculate KV cache size per token?

Multiply: 2 (one K, one V) × number of layers × number of KV heads × head dimension × bytes per element. Everything you need is in the model's config.json.

import json

cfg = json.load(open("config.json"))

layers   = cfg["num_hidden_layers"]                 # 32
kv_heads = cfg.get("num_key_value_heads",
                   cfg["num_attention_heads"])      # 8
head_dim = cfg.get("head_dim",
                   cfg["hidden_size"] // cfg["num_attention_heads"])  # 128
bytes_per = 2                                       # bf16 / fp16

per_token = 2 * layers * kv_heads * head_dim * bytes_per
print(per_token, "bytes =", per_token / 1024, "KiB per token")
# 131072 bytes = 128.0 KiB per token

For Llama 3.1 8B: 2 × 32 × 8 × 128 × 2 = 131,072 bytes. 128 KiB for every single token of every single request.

Notice num_key_value_heads = 8 while the model has 32 attention heads. That's grouped-query attention (GQA): 4 query heads share each K/V head. Llama 2 7B had no GQA, so its KV heads were 32, and the same formula gives 512 KiB per token. GQA is the reason modern 8B models can serve long contexts on consumer cards at all.

A 32,768-token context at 128 KiB per token is 4 GiB. For one request.

Where does the GPU memory actually go?

Weights first, then a reserve for activations and CUDA graphs, and whatever is left becomes the KV cache pool. vLLM measures this at startup and pre-allocates the pool in one go.

Here's the budget on my 24GB card with defaults:

5 GiB ÷ 128 KiB = 40,960 tokens. vLLM splits that into blocks of 16 tokens (2 MiB each in this setup), so that's 2,560 blocks. Exact numbers shift with vLLM version and GPU, which is why you should read your own log rather than trust my table.

vLLM prints all of this at startup. The two lines that matter:

GPU KV cache size: 40,960 tokens
Maximum concurrency for 32,768 tokens per request: 1.25x

That second line is vLLM telling you, politely, that if every request used the full context window, you could run 1.25 of them. I scrolled right past it.

Why did only 4 requests run at once?

Because the pool is shared, and my requests were big. Each RAG call had a ~1,500-token system prompt, ~6,000 tokens of retrieved chunks, and up to ~1,000 tokens of output. Call it 8,500 tokens per request at peak.

40,960 ÷ 8,500 ≈ 4.8. So four requests fit with headroom, and the fifth waits.

max_num_seqs=256 is an upper limit on the batch, not a promise. The scheduler only admits a new request when it has free blocks for it. Compute was idle because a batch of 4 doesn't saturate an 8B model's matmuls. The GPU was memory-bound, not compute-bound.

What happens when the KV cache fills up mid-generation?

vLLM preempts a running request: it frees that request's blocks and later recomputes its whole prompt from scratch. You'll see a warning about preemption in the logs, and the user sees a request that stalls and then takes twice as long.

This is the subtle part. PagedAttention allocates blocks on demand as tokens are generated, not up front for max_tokens. That's what makes it efficient. It also means the scheduler can admit 5 requests that fit right now, then run out of blocks when they all keep generating. Someone gets evicted.

So "it fit at admission" and "it finishes without a redo" are different conditions. If your p95 latency has ugly spikes while p50 looks fine, check for preemption before you blame the network.

How do you increase vLLM concurrency without a bigger GPU?

Shrink the bytes per token, shrink the weights, or stop storing the same tokens twice. Here's the math for each lever on the same card.

1. FP8 KV cache. Store K and V in 1 byte instead of 2.

vllm serve meta-llama/Llama-3.1-8B-Instruct --kv-cache-dtype fp8

128 KiB becomes 64 KiB per token. Same 5 GiB pool now holds ~81,000 tokens, so ~9 of my requests. Check that your GPU and vLLM version support it, and run your own eval set before and after. Quantizing the cache is a precision trade, and you should measure it on your task rather than assume it's free.

2. Quantized weights. An AWQ or GPTQ int4 checkpoint of an 8B model is roughly 5.5–6 GiB instead of 15. That frees around 9 GiB, which all flows into the KV pool: ~14 GiB ÷ 128 KiB ≈ 114,000 tokens, about 13 requests. Combine it with FP8 KV cache and you're near 26. Same caveat: evaluate, don't assume.

3. Prefix caching. My 1,500-token system prompt was identical across requests. With prefix caching, those blocks are computed once and shared.

vllm serve ... --enable-prefix-caching

It's on by default in recent vLLM versions, but it only helps if the shared part is byte-identical and comes first. Put the static system prompt before the retrieved chunks, never after. A timestamp at the top of the prompt kills it.

4. Raise --gpu-memory-utilization. Going from 0.90 to 0.95 adds ~1.2 GiB, roughly 9,600 more tokens. Only do this if nothing else lives on the GPU.

5. Send fewer tokens. Retrieving 12 chunks when 5 would do is a KV cache tax on every request. Cutting 3,000 retrieved tokens per call raises concurrency more than most flags.

Does lowering max-model-len increase concurrency?

No. --max-model-len caps how long one request may be. It does not change how much memory goes to the KV pool, so the token budget stays the same. It changes the "Maximum concurrency" line in the log and lets the server start when the full context window wouldn't fit, but it does not add room.

I tried it first. Dropped it from 32,768 to 16,384, reran the load test, and got the same 4 running, 16 waiting. The pool was the same size. The only thing that changed was the number in the log.

The 30-second capacity check I now run before deploying

def max_concurrent(kv_cache_tokens: int, prompt_tokens: int, output_tokens: int,
                   shared_prefix_tokens: int = 0) -> float:
    per_req = prompt_tokens + output_tokens - shared_prefix_tokens
    usable = kv_cache_tokens - shared_prefix_tokens
    return usable / per_req

# bf16 KV, no prefix sharing
print(max_concurrent(40_960, 7_500, 1_000))          # ~4.8
# fp8 KV, 1,500-token shared system prompt
print(max_concurrent(81_920, 7_500, 1_000, 1_500))   # ~11.5

Plug in the GPU KV cache size from your startup log and your real p95 prompt length, not the average. If the result is below your expected peak concurrency, no amount of batching config will save you.

So why did my 24GB GPU serve only 4 requests at once?

vLLM's concurrency on a single GPU is capped by KV cache memory: every in-flight token costs 2 × layers × kv_heads × head_dim × bytes, which is 128 KiB for Llama 3.1 8B in bf16. After ~15 GiB of weights, a 24GB card has about 5 GiB left, roughly 40,000 tokens shared across all requests, so 8,500-token RAG requests top out at 4 running at once while the rest queue or get preempted. Read the GPU KV cache size line in the startup log, divide by your real per-request token count, and raise the ceiling with an FP8 KV cache, quantized weights, prefix caching, or shorter prompts. max_num_seqs and max-model-len won't move it.

Written by the developer behind Preterview, an interview prep platform.