跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-22AI 評分41

Nestor:用於可擴展預訓練與持續學習的巢狀稀疏性

Nested Sparsity for Scalable Pretraining and Continual Learning

AI 導讀

Nestor 在 attention 中引入巢狀稀疏性,在計算量和稀疏度相同的情況下,語言模型預訓練表現優於傳統 MoE Transformer。Nestor 更有效適應較長上下文,也能更有效率地適應新領域,同時減少對舊領域的遺忘;程式碼和檢查點已在 GitHub 提供。

正文

Published on Oct 22

Authors:

,

Abstract

Nestor introduces nested sparsity in attention to improve conditional computation over standard mixture-of-experts transformers, enhancing pretraining, long-context adaptation, and continual learning.

Mixture-of-experts (MoE) transformers, the standard backbone for modern large-scale deep learning, employ conditional computation: only a sparse subset of model weights is used in each forward pass. In MoE transformers, this sparsity arises in the MoE operator, which generalizes the multi-layer perceptron in the classical transformer. In this work, we explore an alternative mechanism for conditional computation that analogously generalizes the attention operator. The sparsity induced by this mechanism naturally nests (i.e., hierarchically composes) with the sparsity induced by the MoE operator. We show that a transformer employing this nested sparsity mechanism, which we call Nestor, performs better than a conventional MoE transformer at equal compute and equal sparsity in language model pretraining. We also show that Nestor is well-suited for continual pretraining: it can be adapted to longer contexts more effectively than conventional MoEs, as well as adapt more efficiently to new domains while forgetting less on old domains. Furthermore, we demonstrate that these benefits persist as model size, dataset size, and context length grow. Our results demonstrate the potential of alternative conditional computation mechanisms for deep neural network architectures. Code and checkpoints are available on GitHub at https://github.com/Ma-Lab-Berkeley/Nestor .

View arXiv page View PDF Project page GitHub 2 Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.31818 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.31818 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.31818 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co