LightningMoBA:透過動態路由、以 KV 為核心的執行及執行時流水線加速 MoBA
Accelerating Mixture of Block Attention via Dynamic Routing, KV-Centric Execution and Runtime Pipelining
LightningMoBA結合動態路由、KV重用及流水線執行,提升MoBA的GPU注意力計算效率。於五個模型、4K至128K上下文及完整RULER測試中,相較FlashMoBA平均端到端速度提升1.20–1.60×,平均RULER分數由74.42升至75.28。
Sparse attention improves efficiency by reducing unnecessary computation. On GPUs, however, sparse-index construction, irregular memory access, and task scheduling can offset part of those savings. MoBA reduces computation by routing each query to relevant KV blocks, while FlashMoBA further optimizes GPU execution. Building on this foundation, we address two limitations: fixed Top-k budgets do not adapt to different query score distributions, and query-centric task ownership can lead to repeated KV-block loads across tasks, limiting data reuse.
We designed LightningMoBA to jointly optimize routing and GPU kernel execution, with three main improvements over FlashMoBA:
🔹 Dynamic routing: Replace fixed Top-k with a relative threshold, allowing each query to retain a different number of relevant blocks. Sparse indices are generated directly during scoring, without maintaining an ordered Top-k candidate set or training an additional indexer.
🔹 KV reuse: Within each attention head, group queries that access the same KV block so that each task reuses its loaded K/V. Work for heavily accessed blocks is split across multiple tasks to bound the per-task workload.
🔹 Pipelined execution: On Hopper GPUs, overlap data movement and query gathers with Tensor Core computation to reduce stalls.
🚀 Our experiments evaluate both end-to-end performance and the contributions of individual optimizations. Across five models and context lengths from 4K to 128K on the full RULER suite, matching the average retention budget to Top-16 yields 1.20–1.60× average end-to-end speedups per model over FlashMoBA, while the overall mean RULER score increases from 74.42 to 75.28. To separate routing gains from execution-backend improvements, we also compare both routing strategies under the same optimized backend. Across six configurations on two models, dynamic routing at the Top-64 budget improves both speed and quality, achieving 1.16–1.56× end-to-end speedups and 0.85–5.95 percentage-point gains in RULER scores.
We also extend LightningMoBA to training. In a 100-step Qwen3-4B training experiment with 128K context, both routing strategies use the same optimized forward and adapted backward implementations. Dynamic routing achieves 1.23× and 1.60× end-to-end training speedups at Top-32 and Top-64, respectively, while training cross-entropy remains close between the two strategies.
We hope LightningMoBA can further improve MoBA’s training and inference efficiency and provide useful guidance for implementing other dynamic sparse attention methods on GPUs.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co