跳到正文
DEV Community· Sparkle Qwen27b·· 7 小時前AI 評分49

NVFP4 無法節省 VRAM?12GB 手提電腦 GPU 上的量化取捨

NVFP4 Won't Save Your VRAM: Quantization Tradeoffs on a 12GB Laptop GPU

AI 導讀

在 12GB RTX 5070 Ti Laptop 上運行 Qwen3.8-27B 時,NVFP4 峯值 VRAM 僅比 RK4V4 低約 50–70 MiB,差異主要來自 KV pool 配置,而非權重量化格式。作者指出,記憶體不足時應先調低 pool 大小。39 題短上下文測試中,RK4V4 + GDN32 答對 35 題,NVFP4 答對 34 題。

正文

Running a 27B model on a 12GB laptop GPU teaches you to question every assumption about memory — including the most obvious one: that a lower-precision format always saves VRAM.

I run Qwen3 27B as a local agent on an RTX 5070 Ti Laptop (12GB VRAM, i7-14650HX, 32GB RAM, Windows), using NInfer + KVMem tiered KV + MTP3. The repo is Sparkle-Qwen27b-Kvmem-ninfer: https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer

I switched the daily quantization profile from NVFP4 to RK4V4, and the VRAM numbers surprised me. Here is what I measured, and what it means if you are sizing a local setup.

The two profiles

Same model, same machine, same workload shape:

  • Model: Qwen3.8-27B GSQ-RCO IQ3_XXS + MTP3
  • KV budget: DeviceKV 36864 / Retrieve 3840 / HostKV 7168 MiB, xhigh
  • Previous daily profile: NVFP4 — whole-GPU peak 11,168 MiB
  • Current daily profile: RK4V4 + GDN FP32 — whole-GPU peak 11,220–11,236 MiB

So switching to NVFP4 saved roughly 50–70 MiB at the peak. That sounds like a win for NVFP4, until you look closer: at the same pool size, RK4V4 actually uses about 20 MiB less. The difference between the two peaks is the pool configuration, not the weight format.

Why the weights are not the deciding factor

Once a 27B model is already aggressively quantized, the weights are a modest slice of the 12GB budget. The KV cache is the part that scales with context, and the pool is sized to use almost everything available — deliberately. Peak VRAM here is dominated by the DeviceKV pool, not by whether the weights are NVFP4 or RK4V4.

That is why "switch to a cheaper quantization" is bad first advice when you run out of memory on this kind of setup. The lever that actually moves VRAM is the pool size: if you OOM, drop the pool to 38,912 / 38,400 first. Swapping quantization to chase a rescue will not save you.

What the quality check said

The reason RK4V4 is the default anyway is quality, not VRAM. On a 39-question short-context check:

  • RK4V4 + GDN32: 35/39 correct, mean total variation vs BF16 0.0262, 0 prediction flips
  • NVFP4: 34/39 correct, 1 prediction flip

Tool/JSON regression was 20/22 for both — same score, same failures. Long-context results held on the RK4V4 profile: a 110K task finished 12/12, a 218K task 26/26, and a 189,954-token real-content multi-turn run went 24/24. Peak VRAM stayed in the 11,220–11,236 MiB band.

One question and one flip is a small sample, and I would not overclaim from it. But combined with the VRAM finding, the conclusion is clear enough for my setup: NVFP4 costs a little quality and buys essentially no VRAM. The cheapest quantization on paper is not always the cheapest once the KV budget is included.

Honest limits

  • These are measurements from one machine, not a unified benchmark. Different binaries, ports and fixtures across runs.
  • 256K is the logical context capacity (262,144 tokens). Full 256K is not validated; compaction is around 184K, and middle history is retrieved by page selection.
  • The prebuilt engine is Windows x64 / sm120a only. RTX 30/40 need an architecture-matched build.
  • The 232K random-order stress test was unstable across repeats and is not a release claim.

If you run local models on consumer GPUs, I would like to hear your experience: where does your memory actually go — weights, KV cache, or something else entirely?

Repo: https://github.com/CSUNX233/Sparkle-Qwen27b-Kvmem-ninfer

來源:DEV Community · dev.to