跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-17AI 評分35

ECHO:超越輔助式多 token 預測的高效因果自遞歸

ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction

AI 導讀

ECHO 將多視野預測整合至骨幹模型的自遞歸機制,並以動態深度取樣提升預訓練效率及推理草擬能力,毋須輔助模組。預訓練在推理與程式碼任務上提升下游準確度,開銷僅為訓練所有深度的 MTP 一半。推理時,模型可生成超出訓練深度的草稿,並在開放式高熵生成中維持草稿與目標模型的對齊。

正文

Published on Oct 17

Authors:

,

,

,

,

,

,

Abstract

ECHO integrates multi-horizon prediction into the backbone via self-recursion and dynamic depth sampling, improving pretraining efficiency and inference drafting without auxiliary modules.

Existing Multi-token prediction (MTP) methods are auxiliary by design: the multi-horizon loss mostly trains modules attached to the backbone, and each added horizon costs another module and another pass. We propose Efficient Causal Self-Recursion (ECHO), which removes both costs. The backbone's own final layer predicts every horizon, so the supervision lands on the parameters that generate the next token, and Dynamic Depth Sampling (DDS) keeps deeper horizons affordable by drawing the training depth with probabilities that decay along depth. Future prediction thus becomes part of the autoregressive objective rather than a task attached to it, and the gains follow on both sides of it. In pretraining, added depth converts into downstream accuracy where auxiliary modules saturate, concentrating on reasoning and code at half the MTP training overhead of training every depth. At inference, the pretrained model is its own drafter: it drafts beyond the depth it was trained for, and keeps its draft-target alignment on the open-ended, high-entropy generation where post-hoc drafters lose theirs. The advantage survives a doubled token budget, a repeated pretraining run, and supervised finetuning.

View arXiv page View PDF GitHub Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.24766 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.24766 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.24766 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co