跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-05AI 評分37

分佈穩健的混合專家(MoE)訓練

Distributionally Robust Mixture-of-Experts Training

AI 導讀

論文提出 Distributionally Robust MoE Training(DRMoET),以優化高損失路由結果,提升稀疏 MoE 的路由穩健性。在 10.3B 總參數、67B 訓練 tokens 設定下,七項任務平均分由 standard FLAME-MoE 的 0.6625 升至 0.6767,無輔助損失平衡基線則為 0.6431。

正文

Published on Oct 5

·

Submitted by

xt2251

on Oct 12

Authors:

,

Abstract

Mixture-of-Experts (MoE) transformers scale capacity by activating only a few experts per token, but this sparsity creates a hidden reliability problem: when routing is imperfect, load-balanced models may send tokens to experts that are insufficiently trained for the assigned inputs. We propose Distributionally Robust MoE Training (DRMoET), a drop-in objective that treats layer-wise experts as endogenous robustness groups and optimizes high-loss routing outcomes rather than merely equalizing traffic. DRMoET updates a per-layer expert distribution by an entropy-regularized softmax rule on EMA-smoothed, activation-weighted expert losses, strengthening plausible non-top routing paths while preserving standard MoE computation. Under the FLAME-MoE recipe at 746M-total and 10.3B-total scales, DRMoET improves downstream averages over both standard FLAME-MoE and auxiliary-loss-free balancing. At 10.3B total parameters and 67B training tokens, DRMoET improves the seven-task average from 0.6625 to 0.6767, while the auxiliary-loss-free baseline achieves 0.6431. Mechanistic analyses show lower expert-loss variance with nearly unchanged mean loss, 4.3% lower excess loss under forced mid-k misrouting, and improved domain-expert specialization. These results position routing robustness-not only utilization balance-as a practical objective for reliable sparse MoE scaling. Project page and code are available at: https://drmoet.github.io/.

View arXiv page View PDF Project page GitHub 1 Add to collection

Get this paper in your agent:

hf papers read 2610.07207

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 1

TonyTeng/DRMoET

Text Generation •

Updated 5 days ago

•

1

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.07207 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.07207 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co