為何「35GB」量化大語言模型仍會令 80GB GPU OOM：KV Cache、GQA 與 DeepSeek MLA 的 VRAM 計算

Every engineer deploying open-weights models hits the same rite of passage:

You take a 70B parameter model, quantize it to 4-bit (70B * 0.5 bytes ≈ 35 GB), provision a single 80GB A100 or H100 (or two 24GB GPUs), and launch vLLM or llama.cpp with a 64K context window and a small concurrent batch size.

Ten seconds into load testing:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.50 GiB.

How does a "35 GB" model run out of memory on an 80 GB GPU?

Because model weights are only the static floor of LLM memory. Once you factor in quantization scale-block overhead, prefill activation buffers, CUDA/PyTorch workspace reservations, and the silent giant—the Key-Value (KV) Cache—your real VRAM footprint can easily double or triple.

Here is the exact byte-level math behind production LLM VRAM allocation, how Grouped-Query Attention (GQA) and DeepSeek's Multi-Head Latent Attention (MLA) change the scaling curve, and how to size your GPU hardware before paying for cloud instances.

1. The 4 Components of Inference VRAM

At runtime, total GPU memory consumption is governed by four distinct allocations:

Total VRAM = M_weights + M_kv_cache + M_activations + M_runtime_overhead

2. Why 4-Bit Quantization Is Never Exactly 0.5 Bytes/Param

A common rule of thumb says INT4 = 0.5 bytes per parameter. In practice, naive uniform 4-bit quantization destroys model perplexity because weight distributions contain extreme outliers.

Modern quantization formats (AWQ, GPTQ, GGUF K-Quants) group weights into blocks (e.g., 32, 64, or 128 weights) and store higher-precision FP16 scales and zero-points per block, while keeping embedding and output head layers (lm_head) at 16-bit or 8-bit precision:

// Effective bits per parameter (bpw) in production formats
const EFFECTIVE_BITS_PER_WEIGHT: Record<string, number> = {
  FP16: 16.0,      // 2.00 bytes/param
  BF16: 16.0,      // 2.00 bytes/param
  FP8: 8.5,        // ~1.06 bytes/param (channel/tensor scaling factors)
  INT8: 8.5,       // ~1.06 bytes/param
  INT4_AWQ: 4.5,   // ~0.5625 bytes/param (group_size=128 FP16 scales)
  GGUF_Q5_K_M: 5.7,// ~0.71 bytes/param
  GGUF_Q4_K_M: 4.85// ~0.606 bytes/param (super-blocks + unquantized tensors)
};

export function calculateWeightVramGB(paramsBillions: number, format: keyof typeof EFFECTIVE_BITS_PER_WEIGHT): number {
  const bpw = EFFECTIVE_BITS_PER_WEIGHT[format];
  return (paramsBillions * 1e9 * (bpw / 8)) / (1024 ** 3);
}

For a 70.6B model in GGUF Q4_K_M, weight memory alone is ~39.8 GB (GiB)—already 5 GB higher than the back-of-the-envelope 35 GB estimate.

3. The Silent Killer: Exact KV Cache Math

During autoregressive generation, recomputing attention Keys ($K$) and Values ($V$) for all previous tokens at every step would make decoding $O(N^2)$. Instead, inference engines cache $K$ and $V$ tensors across all transformer layers.

Standard Multi-Head Attention (MHA) & Grouped-Query Attention (GQA)

For a model using Grouped-Query Attention (GQA) (such as Llama 3/4, Qwen 3, or Mistral), the KV cache size in bytes is:

KV_Bytes = 2 (K and V)
         × n_layers
         × n_kv_heads
         × head_dim
         × seq_len
         × batch_size
         × bytes_per_kv_element

Let's plug in the real architectural constants for a 70B GQA model (n_layers = 80, n_kv_heads = 8, head_dim = 128):

• Per-token KV footprint (FP16 KV cache, 2 bytes):

  2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes ≈ 320 KB per token

• Now scale to a 64K context window (65,536 tokens) with batch_size = 2:

  320 KB × 65,536 × 2 = 41,943,040,000 bytes ≈ 39.06 GiB!

Notice what just happened:

• Weights (Q4_K_M): 39.8 GiB
• KV Cache (64K context × batch 2 in FP16): 39.1 GiB
• Activations + CUDA Runtime: ~4.2 GiB
• Total VRAM Required: 83.1 GiB ➔ Instant OOM on an 80GB A100/H100!

(And if the model had used classic Multi-Head Attention with 64 KV heads instead of 8 GQA heads, that same KV cache would have required **312.5 GiB!)

4. How DeepSeek's Multi-Head Latent Attention (MLA) Changes the Game

In DeepSeek-V3 and DeepSeek-R1 (671B total MoE parameters, 37B active), keeping standard GQA tensors across 61 layers at 128K context would bottleneck PCIe/NVLink bandwidth.

Instead, Multi-Head Latent Attention (MLA) projects Keys and Values jointly into a compressed low-rank latent vector ($c_t^{KV}$, typically dimension $d_c = 512$) plus a decoupled RoPE key dimension ($d_R = 64$):

export function calculateKvCacheBytesPerToken(arch: {
  attentionType: 'GQA' | 'MLA';
  numLayers: number;
  numKvHeads?: number;
  headDim?: number;
  kvLoraRank?: number; // e.g. 512 for DeepSeek V3/R1
  qkRopeHeadDim?: number; // e.g. 64
  kvBytesPerElem: number; // 2 for FP16, 1 for FP8 KV cache
}): number {
  if (arch.attentionType === 'MLA') {
    const compressedDim = (arch.kvLoraRank ?? 512) + (arch.qkRopeHeadDim ?? 64);
    // MLA caches only the joint latent vector per layer (no 2x K/V multiplier!)
    return arch.numLayers * compressedDim * arch.kvBytesPerElem;
  }

  return (
    2 *
    arch.numLayers *
    (arch.numKvHeads ?? 8) *
    (arch.headDim ?? 128) *
    arch.kvBytesPerElem
  );
}

Instead of caching 2 × n_kv_heads × head_dim elements per layer, MLA caches only 512 + 64 = 576 elements per layer—slashing KV cache memory by ~70% to 80% compared to equivalent GQA models.

5. Three Production Levers to Prevent VRAM OOMs

1. Quantize the KV Cache to FP8 (--kv-cache-dtype fp8 in vLLM): Cutting bytes_per_kv_element from 2 (FP16) to 1 (FP8 E4M3) immediately halves your KV cache footprint with negligible accuracy loss on Hopper/Ada GPUs.
2. Cap --max-model-len to Your Actual P99 Prompt Length: Just because a model supports a 128K context window doesn't mean vLLM should pre-allocate block tables for 128K if your P99 request is under 16K tokens.
3. Enable Chunked Prefill (--enable-chunked-prefill): Without chunked prefill, a single 64K-token prompt spikes intermediate MLP activation tensors during the prefill pass. Chunked prefill bounds peak activation memory to a fixed token budget per step.

🛠️ Interactive Zero-Trust Engineering Calculators

When sizing GPU nodes or auditing token budgets, doing this math by hand across different models and quantization schemes gets tedious fast. I built a set of 100% client-side, zero-trust interactive calculators (no backend calls, no telemetry) that you can use to model your exact workload:

• LLM GPU VRAM & Cloud Cost Calculator: Interactive breakdown of Weights + KV Cache (GQA & DeepSeek MLA) + Activations + Headroom across Llama 4, DeepSeek R1/V3, Qwen 3, Gemma 3, and Mistral (FP16, FP8, INT8, AWQ/GPTQ INT4, GGUF Q4_K_M), with automatic hardware matching (RTX 5090, Mac Studio M4 Ultra, H100, B200) and hourly cloud cost estimates.
• LLM Token & API Cost Calculator: Compare exact token counts and batch/cached API pricing when deciding between self-hosted GPUs vs. frontier APIs (GPT-5.4, Claude 4.6, Gemini 4 Argon, DeepSeek R1).
• Structured Output Schema Converter: Convert JSON/TypeScript schemas directly into strict OpenAI response_format, Gemini responseSchema, and Claude Tool input_schema definitions.

What GPU setup are you currently using for local or production LLM inference, and have you tested FP8 KV cache quantization under long-context loads? Let's compare notes in the comments!