Expert pruning improves decode throughput up to 1.8 with minimal quality loss
Expert pruning improves decode throughput up to 1.8 with minimal quality loss
SlimWise 只在解碼階段剪除專家,保留預填充階段及其完整模型 KV cache,提升 MoE 解碼吞吐量,同時令準確度保持在接近未剪枝模型的水平。Qwen3.6-35B-A3B 剪除 50% 專家時,解碼吞吐量最高提升 1.81×,典型請求的端到端速度約提升 78%。系統無需轉換 KV cache,但較大型模型及更高剪枝比例的表現尚未測試。
Current MoE inference pipelines still stall on expert traffic during batched decoding, limiting latency improvements despite sparsity. SlimWise shows that pruning experts only in the decode phase and reusing prefill KV caches can boost throughput by nearly twofold without perceptible quality loss.
Earlier serving optimizations either prune the full model—including the costly prefilling step—or accept a trade‑off where speed gains come at the cost of generation fidelity. Those approaches sacrifice the rich context built in the prefill stage, leading to noticeable drops in perplexity or output length when the same pruning ratio is applied during both phases.
On Qwen3.6-35B-A3B, SlimWise improves decode throughput by up to 1.81× at 50 % expert pruning with minimal accuracy loss[1]. The system keeps the prefill stage untouched, activates only half of the experts during generation, and still delivers token‑level latency reductions that translate into a roughly 78 % end‑to‑end speedup in typical request patterns.
Without additional training, SlimWise’s KV cache handoff recovers much of the accuracy lost to pruning in many evaluated settings[1]. By feeding the decoder with the full‑model KV cache generated during prefill, the pruned model inherits a high‑quality attention context, so the model’s accuracy remains comparable to the unpruned baseline even when half the experts are disabled.
Because expert pruning preserves the attention architecture and KV cache format, this KV cache handoff requires no KV cache conversion[1]. The unchanged cache layout lets the runtime simply swap the weight mask between phases, eliminating costly data reshaping or additional memory copies that would otherwise erase any latency gains from pruning.
SlimWise’s gains are demonstrated on two MoE backbones and three pruning criteria; performance on larger models or more aggressive pruning ratios remains untested, and the low‑cost distillation stage—while helpful—adds a modest extra training step before deployment. One open question is whether dynamic token‑level expert selection could further tighten the latency–quality curve without requiring a static prune threshold.
Production stacks that serve Mixture‑of‑Experts should replace monolithic decode with a prefill‑decode split and directly feed the full‑model KV cache into a pruned decoder, eliminating the need for costly runtime cache rewrites. Re‑running latency benchmarks under this disaggregated regime will likely reveal hidden speed margins that current end‑to‑end numbers conceal.
References
來源:DEV Community · dev.to