階層式殘差:以極簡方式將殘差寬度與計算寬度解耦
Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width
HiRes 以區塊級寬維度 PostNorm 殘差連接,將殘差寬度與每層計算寬度解耦,提升 Transformer 訓練效率及下游準確度。擬合的 scaling laws 顯示,HiRes 的 FLOPs 效率提升 1.30 倍。以 OLMoE-1B-7B 預訓練 500B tokens 時,訓練 loss 低 0.0208,下游準確度高 1.14 點。
Published on Oct 30
Authors:
,
,
,
,
,
Abstract
Hierarchical Residuals decouple residual width from per-layer compute width via block-level wide PostNorm residual connections, improving training efficiency and downstream accuracy in Transformers.
Residual connections underpin Transformers, but they implicitly tie the residual width carried through depth to the compute width used inside each layer. We introduce Hierarchical Residuals (HiRes), a minimalist architectural change that decouples these dimensions, widening the residual state while keeping per-layer compute width fixed. HiRes groups layers into blocks: standard d-dimensional PreNorm layers operate inside each block, while a wider D-dimensional PostNorm outer residual connects blocks. Learned read/write projections bridge the two spaces at block boundaries, leaving individual Transformer layers unchanged. Across different model sizes and depths, HiRes consistently outperforms vanilla residual addition, with fitted scaling laws indicating a 1.30-fold improvement in FLOPs efficiency. On OLMoE-1B-7B pretraining for 500B tokens, HiRes achieves 0.0208 lower training loss and 1.14-point higher downstream accuracy. HiRes is also competitive with other residual expansion methods such as VWN, mHC and AttnRes with less activation memory and I/O costs. Built from only dense projections and normalizations, it requires neither specialized kernels nor infrastructure-level optimizations. HiRes thus exposes residual width as a practical scaling variable using only standard operators.
View arXiv page View PDF Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.41711 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.41711 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.41711 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co