Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design
Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design
研究發現,前奏—循環區塊—收尾架構提升循環語言模型的上下文擷取與推理表現;Huginn 在 730M 參數、336B tokens 規模下十項基準均勝過 Ouro。在 8BA0.8B(展開後 32BA3B)、以 500B tokens 訓練的 MoE 模型中,Huginn 幾乎所有基準均勝過 Ouro,並較 112 層前饋 MoE 基線少用 75% 常駐參數。
Published on Oct 16
Authors:
,
,
,
,
,
Abstract
Isolating recurrence design axes reveals that a prelude-recurrent block-coda sandwich architecture improves reasoning and context extraction, with advantages transferring to parameter-efficient Mixture-of-Experts models.
Looped language models reuse physical Transformer blocks across logical depth, but established architectures entangle recurrence placement, input injection, and recurrent state organization. We isolate these axes in models trained from scratch at matched parameter scale, logical depth, and token budgets, and then test whether the resulting design principles transfer to resident- and active-parameter-matched Mixture-of-Experts (MoE) models. The prelude-recurrent block-coda sandwich architecture consistently improves performance on context extraction and reasoning-intensive tasks. Input injection primarily enhances knowledge retrieval and context utilization, whereas random initialization and separate H/L recurrent states provide mixed or negative returns. Taken together, these results favor the Huginn-style design: at the 730M-parameter and 336B-token scale, the complete Huginn architecture outperforms Ouro across all ten benchmarks. The advantages of Huginn over Ouro transfer to MoE models. At the 8BA0.8B (32BA3B when unrolled) scale trained on 500B tokens, Huginn outperforms Ouro on nearly all benchmarks spanning knowledge, reasoning, and coding, while also exhibiting more balanced expert routing. Under matched training and inference FLOPs, the Huginn-style model uses 75% fewer resident parameters than a 112-layer feedforward MoE baseline. Despite its smaller physical parameters, it outperforms the feedforward baseline on DROP and GSM8K, matches it on MATH500, and remains competitive on the other benchmarks.
View arXiv page View PDF Project page Add to collection
Get this paper in your agent:
hf papers read 2610.23036
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.23036 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.23036 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.23036 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co