Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
InfiLoop 提出循環原生殘差連接,讓循環式 Transformer 在增加測試時迭代時保留有用的中間狀態。7M 參數的 InfiLoop 模型在 Sudoku-Extreme 達到 97.9% exact accuracy,在 ARC-AGI-2 達到 13.6% pass@2。InfiLoop 在 Sudoku-Extreme 上迭代超過 20,000 個有效步驟後仍持續提升。
Published on Oct 8
Authors:
,
,
,
,
Abstract
In this paper, we argue that looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows. We observe that increasing loop iterations can reduce reasoning accuracy: noisy state updates overwrite correct intermediate deductions and even undo completed solutions. This leaves subsequent iterations to recover lost information from an already degraded representation: once an error arises in an earlier loop, often as a result of long-range propagation through the recurrence, later loops find it difficult to correct. In this paper, we introduce InfiLoop, a loop-native residual connection that learns which past computations to retain and how much to accept from each new update. InfiLoop combines content-based weighting with learned temporal decay to maintain a running summary of recurrent states. An exact streaming recurrence keeps its persistent aggregation memory constant as the loop count grows. The resulting adaptive update suppresses unreliable proposals and preserves useful intermediate states. Across extensive reasoning tasks, a 7M-parameter InfiLoop model outperforms existing recursive architectures, reaching 97.9% exact accuracy on Sudoku-Extreme, and 13.6% pass@2 on ARC-AGI-2. Notably, on Sudoku-Extreme, InfiLoop continues to improve with test-time looping beyond 20,000 effective steps, showing that added depth translates directly into stronger reasoning. Our code is available at https://github.com/pixeli99/InfiLoop.
View arXiv page View PDF GitHub Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.11570 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.11570 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.11570 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co