跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 5 天前AI 評分38

TPHC:以張量並行分組更新改進 Hyper-Connections

TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates

AI 導讀

TPHC 結合張量並行分片與 Hyper-Connections 殘差流,降低語言模型損失並提升零樣本推理表現。8B 模型採四條流時,訓練時間較 mHC 多 9%;流數由四增至十六,訓練時間僅多 19%。1.1B 模型以 100B tokens 訓練後,驗證損失較同流數 mHC 低 0.7-1.0%,六項常識推理基準的平均零樣本準確率高 0.3-0.7 個百分點。

正文

Published on Nov 17

·

Submitted by

whywhy

on Nov 18

Authors:

,

,

,

,

,

Abstract

TPHC combines tensor-parallel shards with hyper-connection residual streams to enable more independent stream updates, improving language model loss and zero-shot reasoning with modest training overhead.

Hyper-Connections (HC) expand the residual stream of a Transformer into N parallel streams, and manifold-constrained HC (mHC) stabilizes them at scale. However, mHC benefits little from more than four streams, partly because each sublayer writes a single output to all streams, limiting how differently they can be updated. In this work, we propose TPHC, which integrates tensor-parallel (TP) shards with HC streams to promote greater independence across streams. Specifically, we route a separate mixture of the residual streams to each TP shard and keep the individual shard outputs separate during write-back, so each stream can receive a different mixture of those outputs. This design makes the residual-stream updates more expressive without adding parameters or FLOPs to the attention or FFN sublayers. With native support for stream sharding under TP, TPHC requires only 9% more training time than mHC on an 8B model when both use four streams. Increasing the number of streams from four to sixteen raises its training time by only 19%. On a 1.1B model trained on 100B tokens, TPHC reduces validation loss by 0.7-1.0% and raises average zero-shot accuracy across six commonsense reasoning benchmarks by 0.3-0.7 percentage points relative to mHC at matched stream counts Nin{4,8,16}. Our code is available at https://github.com/wuyou2002/TPHC.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2611.23026

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.23026 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.23026 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.23026 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co