在 NVIDIA DGX Spark 上提升 MoE 推理速度的 3 種方法

3 Ways to Boost MoE Inference on NVIDIA DGX Spark

AI 導讀

NVIDIA Technical Blog 提出在 NVIDIA DGX Spark 上提升 MoE 推理速度的 3 種方法，並以 Inclusion AI 的 Ling-3.0-flash 評估為例。批次大小為 1 時，效能分析顯示解碼受記憶體頻寬限制，讀取模型權重是主要耗時來源；MoE 計算和輸出層投影是兩個突出的運算環節。

正文

Abstract green computing cube with layered panels, performance icons, and streams of data on a black background Generating tokens faster starts with identifying what limits inference performance. In the batch-size-1 evaluation of Ling-3.0-flash from Inclusion AI on NVIDIA...

Generating tokens faster starts with identifying what limits inference performance. In the batch-size-1 evaluation of Ling-3.0-flash from Inclusion AI on NVIDIA DGX Spark, profiling showed that decoding was memory-bound: reading model weights drove execution time. Two operations stood out—the mixture-of-experts (MoE) computation and the output-layer projection that produces scores across the model’…

Source