Meta Expert:跨層專家共享提升 MoE 的儲存與檢索效率
Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
研究比較 MoE 的跨層共享 Meta Expert 與 Local Expert,發現 Meta Expert 可減少參數冗餘,並降低交叉熵損失、提升下游任務準確度。在啟動數量及專家粒度固定時,使用 70–80% 的專家即可媲美 Vanilla MoE 的效能。理論分析亦顯示,Meta Expert 可提供更靈活的殘差路徑,提升資訊檢索效率。
Published on Oct 20
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Shared meta-experts across layers reduce redundancy and improve efficiency compared to layer-specific local experts in mixture-of-experts models.
As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged Local Expert in existing mainstream models, we hypothesize that inter-layer shared Meta Expert hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, our model matches the performance of Vanilla MoE using only 70--80\% of the experts, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2610.27598
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.27598 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.27598 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.27598 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co