跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-09-28AI 評分34

ReSPO:針對離策略學習梯度飢餓問題的重塑序列策略最佳化

ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning

AI 導讀

ReSPO 針對離策略學習中的梯度飢餓問題,提出以平滑雙分支序列層級核函數取代裁剪的最佳化方法。在密集型及 MoE Qwen3 模型上,ReSPO 加快早期最佳化、提升最終訓練分數,並在重用 rollout 時取得較高的留出基準測試表現。

正文

Published on Sep 28

Authors:

,

Abstract

Reinforcement learning from verifiable rewards (RLVR) frequently reuses rollouts across multiple policy updates, increasing the mismatch between the current policy and the data-generating policy. We identify a sign-dependent gradient starvation problem in clipped policy optimization: clipping suppresses under-generated positive responses at the low-importance-weight tail while permitting severely over-generated negative responses to dominate the high-weight tail. To address this, we propose ReSPO (Reshaped Sequence Policy Optimization), which replaces clipping with a smooth, two-branch sequence-level kernel derived from an α-divergence variational objective and an exponential variance-control tilt. The positive branch preserves a nonzero gradient weight for under-generated positive responses, while the negative branch suppresses heavily over-generated negative responses. We demonstrate that ReSPO effectively learns from long positive reasoning trajectories during early training, even when accumulated policy drift relegates them to the low-importance-weight tail. On dense and MoE Qwen3 models, ReSPO accelerates early optimization, improves final training scores, and achieves higher held-out benchmark performance under a rollout reuse, validating our approach on importance-weight tail control in off-policy learning.

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2609.35433

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2609.35433 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2609.35433 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2609.35433 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co