跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 27 天前AI 評分36

DTCA:為擴散語言模型強化學習進行去噪 token 信用分配

DTCA: Denoising Token Credit Assignment for Reinforcement Learning in Diffusion Language Models

AI 導讀

DTCA 透過衡量 token 提交後其餘遮罩位置的預測信心改善,估算 token 貢獻,並按詞性分類訊號,為擴散語言模型的強化學習進行 token 級信用分配。DTCA 亦按策略更新引起的 token 級熵變化校準學習訊號;在 LLaDA-8B-Instruct 上,MATH500 分數最高提升 9.0 分,GSM8K 最高提升 5.3 分。

正文

Abstract

DTCA improves reinforcement learning for diffusion language models by assigning token-level credit via confidence improvements and entropy calibration, boosting mathematical reasoning.

Diffusion language models (DLMs) generate text through iterative denoising, enabling bidirectional context modeling and parallel token refinement. Recent studies have applied reinforcement learning (RL) to DLMs and improved their reasoning performance. However, the iterative denoising process makes credit assignment more challenging, as final outcome rewards provide limited information about how different tokens contribute to the final output. In this paper, we propose Denoising Token Credit Assignment (DTCA), a framework for token-level credit assignment in reinforcement learning for diffusion language models. DTCA estimates token contributions by measuring the improvement in prediction confidence of remaining masked positions after token commitment and groups these signals according to part-of-speech (POS) categories. These contribution weights are used to select informative denoising timesteps and assign token-level learning signals during RL training. DTCA further calibrates these learning signals according to token-level entropy changes induced by policy updates. Experiments on LLaDA-8B-Instruct show that DTCA consistently improves mathematical reasoning performance across different generation budgets, with gains of up to 9.0 points on MATH500 and 5.3 points on GSM8K over the base model.

View arXiv page View PDF Project page GitHub 3 Add to collection

Get this paper in your agent:

hf papers read 2610.35201

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.35201 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.35201 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.35201 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co