跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 25 天前AI 評分45

多 Token 預測的正確做法:Soft-MTP 移除有噪梯度殘差

Multi-Token Prediction Done Right

AI 導讀

Soft-MTP 透過 straight-through KL 梯度恢復技巧,移除多 Token 預測(MTP)訓練中的噪聲梯度殘差,只反向傳播 KL 自蒸餾訊號。

正文

Published on Oct 28

Authors:

Abstract

Soft-MTP removes a noisy gradient residual from multi-token prediction via a straight-through KL recovery trick, improving perplexity, speculative decoding, and reasoning without added cost.

Multi-Token Prediction (MTP) is now a common ingredient in language-model pre-training, as it improves downstream performance and enables faster inference through speculative decoding. We identify a flaw in standard MTP training, in which a residual component of the MTP gradient interferes with both next-token and multi-token learning, increasingly so with scale. Specifically, the MTP cross-entropy gradient can be decomposed exactly into a KL self-distillation term that pulls each MTP head toward the model's own next-token prediction (NTP) distribution and improves speculative decoding, and a residual that has a low signal-to-noise ratio and buries the useful KL signal. We introduce the KL gradient-recovery trick: when two distributions share a linear output head, the gradient of their KL divergence can be recovered exactly by combining their cross-entropy gradients through straight-through gradient routing. Unlike minimizing the KL directly, this trick requires no vocabulary-sized logit matrices and is compatible with standard fused cross-entropy kernels. This enables Soft Multi-Token Prediction (Soft-MTP), a one-line code change to MTP that removes the noisy residual and backpropagates only the KL self-distillation signal by reusing the existing NTP and MTP cross-entropy computations. Soft-MTP supports both DeepSeek- and Meta-style MTP, adds no parameters, and incurs no measurable overhead. From 167M to 1.2B parameters, Soft-MTP improves over standard MTP with lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy at every scale, lower MTP perplexity from 689M onward, and stronger math and code reasoning after fine-tuning at 1.2B. Soft-MTP also matches, if not exceeds, direct KL divergence minimization on most metrics. Soft-MTP is, in effect, a free lunch for MTP optimization. Our code can be found at https://github.com/JaydenTeoh/Soft-MTP.

View arXiv page View PDF GitHub 1 Add to collection

Get this paper in your agent:

hf papers read 2610.38331

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 12

JaydenTeoh/ntp-167m-model

Text Generation •

0.2B •

Updated about 1 hour ago

•

1

JaydenTeoh/hard-mtp-167m-model

Text Generation •

0.2B •

Updated about 1 hour ago

•

1

JaydenTeoh/soft-mtp-167m-model

Text Generation •

0.2B •

Updated about 1 hour ago

JaydenTeoh/hard-mtp-332m-model

Text Generation •

0.4B •

Updated about 2 hours ago

Browse 12 models citing this paper

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.38331 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.38331 in a Space README.md to link it from this page.

Collections including this paper 1

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co