如何用 llama.cpp 加速 Qwen3.8-27B:NVIDIA 草稿模型未必勝過 MTP
How to Speed Up Qwen3.8-27B in llama.cpp (NVIDIA's Draft Model May Lose to MTP)
Alan West 分析如何以 nvidia/Qwen3.5-4B-Qwen3.8-27B-Draft 草稿模型或 Qwen3.8-27B 內置 MTP,在 llama.cpp 加速 Qwen3.8-27B;他指出單張消費級 GPU 上,草稿模型未必更快。
NVIDIA just shipped a speculative decoding draft model for Qwen3.8-27B, and the model card doesn't contain a single tokens-per-second number. It has acceptance figures, a comparison against Qwen's built-in MTP head, and a note that it was tested on a Groq 3 LPU. If you run the 27B on a 24 GB consumer card, you have to work out the speedup yourself.
So I read the card, the GGUF quants, and the current llama.cpp and vLLM docs to figure out the VRAM cost, the flags each runtime needs, and when the draft ends up making things slower. Short version: on one consumer GPU, the MTP head already inside your 27B GGUF may well beat it.
What a draft model does for Qwen3.8-27B
On a single GPU, decoding is memory-bound. Each token means streaming all 27B weights through the GPU, and the compute mostly sits idle. Speculative decoding uses that idle compute. A small model guesses the next N tokens cheaply, then the big model checks all N in one forward pass. Every guess it agrees with is a token you didn't have to pay a full pass for. Rejected tokens get resampled, so the output distribution stays the same as the target's.
nvidia/Qwen3.5-4B-Qwen3.8-27B-Draft is a full language model, not a head. It has about 4.21B parameters, uses the Qwen3_5ForCausalLM architecture and the target's tokenizer and vocabulary, and was trained with NVIDIA Model Optimizer (nvidia-modelopt v0.48.0rc0). The training data was 199,998 prompts from Nemotron-Post-Training-Dataset-v2, with responses generated by Qwen3.8-27B itself, and the draft learned from the target's top-20 output distribution at each position. That's why it should agree with the 27B more often than a stock Qwen3.5-4B would.
The card lists it under the NVIDIA Open Model Agreement as "ready for commercial or non-commercial use". It went up on November 17, 2026.
Where the 46% comes from
Everything below comes from NVIDIA's SPEED-Bench table on the model card, at block size 16:
| Draft | Params | Qualitative | Throughput-32k |
|---|---|---|---|
| NVIDIA Qwen3.5-4B draft | 4.21B | 5.58 | 5.11 |
| Qwen3.5-0.8B draft | 0.75B | 3.92 | 3.60 |
| Llama-3.2-1B draft | 1.48B | 3.42 | 3.07 |
| Qwen3.8-27B built-in MTP head | 0.42B | 3.81 | 3.86 |
The card calls these "acceptance rates", but a rate can't be 5.58. The card's own definition is "tokens returned per target step", so read them as mean accepted lengths. 5.58 / 3.81 is where the 46% headline over the MTP head comes from. On the long-context subset the gap shrinks to 5.11 vs 3.86, about 32%. By category, RAG gets the best result (6.58) and roleplay the worst (4.41), which is what you'd expect: drafts do well when the answer copies from the context and badly when the output is high-entropy.
The catch is that the 4B draft is also ten times the size of the MTP head, and acceptance length is only half of the speedup formula. The other half is what each draft token costs you.
Napkin math: will it beat MTP on your GPU?
The usual approximation is speedup ≈ τ / (1 + γ·c), where τ is accepted length, γ is tokens drafted per step, and c is the cost of one draft pass relative to one target pass. On a memory-bound GPU, c is roughly the ratio of weight sizes.
With bartowski's GGUFs, the draft at Q4_K_M is 2.72 GB and the 27B at Q4_K_M is 17.44 GB, so c ≈ 0.16. At NVIDIA's block size of 16, the draft costs about 2.5 target passes per step, which gives 5.58 / 3.5 ≈ 1.6x. The MTP head is about 0.42B against 27B, so c ≈ 0.016, which gives 3.81 / 1.25 ≈ 3x on paper.
This ignores verification cost, the 248,320-token vocabulary, and the checkpoint overhead I cover below, so don't treat these as benchmarks. It also flatters MTP: per the card, the head has no embedding or output head of its own and reads the target's, another 2.54B parameters, so each MTP draft step really costs more than 0.42B suggests. The rough numbers do explain why NVIDIA tested on an LPU, though: sixteen-token blocks make sense when draft compute is nearly free. On a 4090 it isn't, so you'd run a shorter draft, which also lowers τ. My bet is that the MTP head you already have stays competitive on a single consumer GPU, and the 4B draft earns its VRAM on RAG-style workloads and on hardware where draft passes cost almost nothing. I'd test both; commands for each are below.
How much VRAM the draft costs
Draft sizes from bartowski's GGUF repo and target sizes from the Qwen3.8-27B GGUF repo:
| Target quant | Draft quant | Weights total | Fits on |
|---|---|---|---|
| IQ3_XXS (12.32 GB) | IQ4_XS (2.43 GB) | 14.75 GB | 16 GB, barely any context left |
| IQ4_XS (15.48 GB) | IQ4_XS (2.43 GB) | 17.91 GB | 24 GB with room for context |
| Q4_K_M (17.44 GB) | Q4_K_M (2.72 GB) | 20.16 GB | 24 GB, moderate context |
| Q5_K_M (20.92 GB) | Q4_K_M (2.72 GB) | 23.64 GB | Not on 24 GB once you add context |
Those totals are weights only. The draft also keeps its own KV cache and recurrent state. Don't quantize the draft down to Q2 to save a gigabyte, because a worse draft gets rejected more and the speedup goes with it. The same goes for the target: the 5060 Ti write-up ran a 27B at IQ3_XXS and gives "aggressive quantization breaks alignment" between draft and target as one reason its draft runs failed. That makes the 16 GB row above the riskiest one.
llama.cpp: the -md command
bartowski made the draft GGUFs with llama.cpp b12412, so use that build or newer. Many older tutorials use --draft-max and --draft-min, which have been removed; current builds tell you to use --spec-draft-n-max and --spec-draft-n-min instead.
llama-server \
-m Qwen3.8-27B-Q4_K_M.gguf \
-md Qwen3.5-4B-Qwen3.8-27B-Draft-Q4_K_M.gguf \
--spec-type draft-simple \
--spec-draft-n-max 8 \
--spec-draft-n-min 2 \
-ngl 99 -ngld 99 \
-c 32768 -fa on
Notes from the server README:
-
--spec-typedefaults tonone, so setdraft-simpleexplicitly. -
--spec-draft-n-maxdefaults to 3. Start around 8 and adjust in both directions. - If you sample at temperature 1.0 like NVIDIA did (top-k 20, top-p 0.95), add
--spec-draft-sampling probabilistic. The default isgreedy. - Don't combine it with an n-gram mode. The docs say "the draftless decoding has higher precedence."
- To compare against MTP, drop
-mdand use--spec-type draft-mtp. bartowski's 27B quants keep the MTP layers (stored at Q4_0 in the imatrix quants).
vLLM: the speculative-config
The card says the checkpoint loads in vLLM but gives no command. Going by the vLLM speculative decoding docs:
vllm serve Qwen/Qwen3.8-27B \
--speculative-config '{
"method": "draft_model",
"model": "nvidia/Qwen3.5-4B-Qwen3.8-27B-Draft",
"num_speculative_tokens": 8,
"draft_sample_method": "probabilistic"
}'
Use draft_tensor_parallel_size for multi-GPU, not tensor_parallel_size, which the docs say is not a valid key here. Draft models aren't supported at all in vllm<=0.10.0. Earlier this year, a user on the vLLM forum got a tensor size mismatch (5120 vs 4096) when using a stock Qwen3.5-4B as a draft for the 27B. I haven't seen anyone confirm that NVIDIA's checkpoint avoids it, so pin your vLLM version and check the startup log.
When the draft makes you slower
- Rolling back the hybrid layers is expensive. Qwen3.8-27B has 3 Gated DeltaNet layers for every Gated Attention layer, and the recurrent state can't be trimmed like a KV cache when a draft is rejected. llama.cpp handles this with checkpoints (PR #19493, merged April 2026), and the author says plainly they are "not as fast as
llama_memory_seq_rm". Every partial accept means restoring a checkpoint and running a shorter batch again. One tester in that PR saw "well above 30%" gains on code and about 10% slower generation on ordinary text. - Short outputs don't pay it back. A 40-token answer doesn't give the draft enough tokens to make up for loading it.
- High-entropy output gets rejected a lot. Roleplay is NVIDIA's own worst category, and creative writing behaves the same way.
- If adding the draft pushes target layers or KV cache off the GPU, you lose far more than speculation gets back.
- It has already happened to someone. The 5060 Ti test above ran Qwen3.6-27B at IQ3_XXS with Qwen3.5-0.8B as the draft and got 22 t/s in llama-server against a 31 t/s baseline, and the author flags that run's 100% acceptance as fake. The best run with real acceptance (ik_llama.cpp, 26% accepted) managed 19 t/s. That was before NVIDIA's draft existed and on an older build, but it shows how easily you can end up slower.
Before you keep it on, run the same prompt three times with and without -md and compare generation t/s. Then do it again with draft-mtp. Use prompts from your real workload for this, and keep whichever setup wins.
Has speculative decoding actually sped up your local setup, or did the draft just eat your VRAM? Post your GPU, quants and t/s numbers. I'd especially like to see someone show the 4B draft beating the MTP head on a single 24 GB card.
Sources
- nvidia/Qwen3.5-4B-Qwen3.8-27B-Draft model card
- bartowski/Qwen3.5-4B-Qwen3.8-27B-Draft-GGUF
- bartowski/Qwen3.8-27B-GGUF
- Qwen/Qwen3.8-27B model card
- llama.cpp server README and speculative decoding docs
- llama.cpp PR #19493: server speculative checkpointing
- vLLM speculative decoding docs
- vLLM forum: Qwen3.5-27B-FP8 speculative decoding
- Qwen 27B dense speculative decoding on an RTX 5060 Ti
來源:DEV Community · dev.to