跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-20AI 評分44

Specter:以量化支援的多步驟推測卸載加速 MoE 推理

Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

AI 導讀

Specter 結合量化推測解碼與專家卸載,加速單 GPU 上的 MoE 推理,並透過預測未來專家用量,提前安排專家傳輸與執行。單請求推理下,Specter 的每輸出 token 時間(TPOT)平均加速 1.50x,最高達 1.88x。

正文

Published on Oct 20

·

Submitted by

Liu

on Oct 21

Authors:

,

,

,

,

,

,

Abstract

Specter accelerates single-GPU mixture-of-experts inference by combining quantized speculative decoding with expert offloading to hide PCIe latency and improve throughput.

Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request inference, Specter achieves an average 1.50x (up to 1.88x) speedup in time per output token (TPOT) over the state-of-the-art system. Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware. Our code is available at https://github.com/wwwwwf/specter_asplos.

View arXiv page View PDF Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2511.14102 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2511.14102 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2511.14102 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co