跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-09-28AI 評分40

How to Loop MoE: Flatten the Experts, Untie the Attention

How to Loop MoE: Flatten the Experts, Untie the Attention

AI 導讀

Foil 提出在固定參數量、專家計算量及有效深度下,展平專家層、增加每層專家數和循環次數,並為每次循環使用獨立注意力參數。訓練至 20B tokens 時,所有 Foil 配置的預訓練 loss 均低於基線;至 100B tokens 時,展平程度越高,loss 越低,完全展平配置比基線低 0.012 nat。

正文

Looped Transformers reuse the same block across multiple passes, while sparse Mixture-of-Experts models store many experts but activate only a few for each token. These two ideas are naturally complementary: every new pass gives a token another routing decision, allowing it to reach different experts and expert combinations without adding expert parameters. But this raises an open design question: under a fixed parameter and compute budget, how should experts be distributed across layers and passes; and which components should be shared across passes?

Our new preprint, “How to Loop MoE: Flatten the Experts, Untie the Attention,” answers this question with Foil.

𝗢𝘂𝗿 𝗶𝗱𝗲𝗮: Flatten the experts. Loop more. Untie the attention.

At each flattening step, Foil uses:
→ half as many expert layers
→ twice as many experts per layer
→ twice as many recurrent passes
→ separate attention parameters for each pass
while keeping total parameters, expert compute per token, and effective depth fixed.

We go from:
8 experts × 8 layers × 2 passes to:
64 experts × 1 layer × 16 passes

𝗥𝗲𝘀𝘂𝗹𝘁𝘀:
→ Every Foil configuration achieves lower pretraining loss than the baseline at 20B tokens.
→ 𝗔𝘁 𝟭𝟬𝟬𝗕 𝘁𝗼𝗸𝗲𝗻𝘀, 𝗹𝗼𝘀𝘀 𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝘀 𝗺𝗼𝗻𝗼𝘁𝗼𝗻𝗶𝗰𝗮𝗹𝗹𝘆 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲 𝗱𝗲𝗴𝗿𝗲𝗲 𝗼𝗳 𝗳𝗹𝗮𝘁𝘁𝗲𝗻𝗶𝗻𝗴 𝗮𝘁 𝗺𝗮𝘁𝗰𝗵𝗲𝗱 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀 𝗮𝗻𝗱 𝗰𝗼𝗺𝗽𝘂𝘁𝗲. The fully flattened Foil ends 0.012 nat below the baseline, with downstream accuracy on par or better.
→ Untying attention becomes increasingly valuable as the model gets flatter. At the fully flattened shape, it lowers loss by 0.049 nat and improves mean accuracy on three representative downstream tasks by 3.3 points over tied attention at the same shape, with no additional compute.

→ More experts per layer make additional loops more useful; and more loops make additional experts more useful. The two amplify each other.
→ Load balance alone is not enough; it should be read together with routing confidence. In our experiments, routing confidence moves with looping gain, and its per-pass peak may signal diminishing returns from further looping or flattening.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co