解析個人化 MoE 大語言模型中專家的角色
Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization
研究提出以稀疏 MoE 大語言模型進行個人化,並識別承載不同個人化特徵的專家。模型表現可媲美 dense LLM,只啟用部分參數,且在活躍參數相近時多義性較低。因果乾預顯示,個人化潛在表徵足以重現模型行為;提高相關專家的啟用程度,可令非個人化模型趨近個人化模型。
Published on Oct 15
Authors:
,
Abstract
Personalizing sparse Mixture-of-Experts language models enables localized, interpretable control over user-specific behaviors through specialized expert routing and learned personalized latents.
Personalization of Large Language Models (LLMs) is increasingly important for user-centric applications, yet prior methods mainly adapt dense LLMs, whose polysemantic neurons, each encoding multiple unrelated concepts, hinder localizing the components behind personalization, and thus understanding and controlling it. We propose to personalize Mixture-of-Experts (MoE) LLMs, which sparsely route each token to a few specialized experts, reducing polysemanticity and enabling such localization. We adapt three open-weight MoE LLMs on a large-scale personalization dataset and train crosscoders on paired activations of the personalized and non-personalized models to learn interpretable personalized latents, from which we identify the experts encoding distinct personalization features. The personalized MoE LLMs perform competitively with dense LLMs while activating only a fraction of their parameters, exhibit lower polysemanticity than dense LLMs with similar active parameters, and, within the same model family, yield personalized latents that are less shared with the non-personalized model. Causal interventions show that these latents suffice to recover the personalized model's behavior and that boosting their associated experts moves the non-personalized model toward it, revealing distinct routing paths for different personalization features.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2610.22127
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.22127 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.22127 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.22127 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co