跳到正文
Cohere Labs:官方研究博客·· 5 天前AI 評分44

為何稀疏 MoE 模型上的投機解碼比預期更有效

Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models

AI 導讀

研究評估了 10 個稀疏 MoE 模型,涵蓋 30B--1T 參數及 8--512 個專家,發現近期細粒度模型在低批次大小下的投機解碼加速效果更高,打破過往預測。驗證成本還受草稿 token 間的路由重疊及 GPU 執行特性影響;研究提出的 IPF 框架預測獨立啟用專家數量時,誤差低於 4%。

正文

Oct 28, 2026

Speculative decoding (SD) accelerates autoregressive inference by leveraging the memory bound nature of decoding, verifying multiple draft tokens in a single forward pass. For mixture-of-experts (MoE) models, prior work predicts a concave speedup-batch size relationship, with gains peaking at intermediate batch sizes due to expert saturation.

Authors

Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli

Abstract

Speculative decoding (SD) accelerates autoregressive inference by leveraging the memory bound nature of decoding, verifying multiple draft tokens in a single forward pass. For mixture-of-experts (MoE) models, prior work predicts a concave speedup-batch size relationship, with gains peaking at intermediate batch sizes due to expert saturation. We evaluate this across 10 sparse MoE models (30B--1T parameters, 8--512 experts) and find recent fine-grained models break this trend, showing higher speedup at low batch size. We find verification cost is governed not only by activated-expert bytes, but also by temporal overlap across draft tokens and GPU execution dynamics. At low batch size, MoE layers in fine-grained models account for a much smaller share of runtime than of weights; on H100, kernel profiling attributes this mainly to higher bandwidth utilization of MoE kernels relative to attention, with L2 absorbing repeated reads of co-routed experts. Across draft tokens, routing overlap further reduces the unique experts activated during verification relative to independence assumptions. We model this effect using a maximum-entropy Iterative Proportional Fitting (IPF) framework that predicts unique expert counts from routing-overlap statistics with less than 4\% error, and substantially narrows the gap to real-model curves and captures the qualitative low-batch trend from synthetic routing.

Related works

來源:Cohere Labs:官方研究博客 · cohere.com