跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-14AI 評分46

DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections

DAGFormer: Global Topology Prediction for Transformer Cross-Layer Connections

AI 導讀

DAGFormer 透過聯合訓練的拓撲預測器,全局規劃 Transformer 的跨層連接,降低困惑度並提升下游任務表現。在 75M 至 1B 參數的配對 OLMo-2 測試中,其 WikiText-2 困惑度降低 17% 至 25%;1B DAGFormer 使用 5B tokens 訓練,表現超越使用 10B tokens 訓練的 vanilla Transformer。

正文

Published on Oct 14

Authors:

,

Abstract

DAGFormer improves Transformer efficiency by using a jointly trained topology predictor to plan cross-layer connections globally, reducing perplexity and boosting downstream task performance across model scales.

We propose DAGFormer, a Transformer language model whose cross-layer connections are planned by a jointly trained topology predictor. Beyond the residual connections in a standard Transformer, DAGFormer adds direct connections from earlier layers to later ones, and the predictor controls the strength of every connection based on input tokens. Unlike existing cross-layer routing architectures, which set each layer's connections separately, DAGFormer plans the routing of the whole model at once. On matched OLMo-2 pairs from 75M to 1B parameters, DAGFormer lowers WikiText-2 perplexity by 17% to 25% over the vanilla Transformer, and a 1B DAGFormer trained on 5B tokens outperforms the vanilla model trained on 10B. Against four routing baselines with the same data and budget, it reaches the lowest perplexity at 75M, 300M and 1B. The gain carries over to 13 commonsense, science, reading and math evaluation tasks (2.0 to 2.5 points higher mean accuracy at 300M and 1B) and to instruction finetuning (63% to 70% AlpacaEval win rate against the vanilla Transformer, leading every routing baseline), and under structured pruning DAGFormer degrades the least in five of six settings.

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.19625

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.19625 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.19625 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.19625 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co