跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-09-28AI 評分40

學會引導,藉引導觀察:以可訓練向量揭示大語言模型中 RLVR 的幾何結構

Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors

AI 導讀

研究者在 5 個大語言模型及 6 項可驗證獎勵任務上研究 RLVR,發現其增益與激活空間中的低維有效流形相關。他們提出即插即用框架 Alpha-Stabler,監察主成分子空間侵入並移除反向傳播時激活梯度中的相關分量,使訓練穩定達 2,000 步,並持續提升 RL 增益。

正文

Published on Sep 28

Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Reinforcement learning (RL) has become a key paradigm for enhancing the reasoning of large language models, yet the high dimensionality of parameter updates makes its training dynamics hard to analyze. We study reinforcement learning with verifiable rewards (RLVR) and use vector steering to identify a low-dimensional effective manifold in activation space associated with RL-induced gains. We uncover two geometric properties. (1) Effective Manifold Capacity: the capacity needed to reproduce RL gains can be very small but is not infinitely compressible; at extremely low capacity, intervention dimensionality and input-dependent expressiveness become key constraints, and this requirement varies with injection depth. (2) Control Manifold Separation: effective control directions lie mainly in the low-variance complement of the activation principal subspace. Within a task and base model, the learned geometry stays largely consistent across training configurations, and across tasks geometric alignment correlates with capability transfer. Experiments on 5 LLMs and 6 verifiable-reward tasks support these findings. We then propose Alpha-Stabler, a plug-and-play framework with a Predictor that monitors principal-subspace intrusion for early collapse warnings, and a Controller that removes the principal-subspace component of activation gradients during backpropagation while preserving the orthogonal complement. Alpha-Stabler stabilizes training for 2,000 steps and consistently improves RL gains, offering practical insights for robust post-training. Code: https://github.com/caiyuchen-ustc/On_Policy_Vector_Training

View arXiv page View PDF GitHub 43 Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2609.34344 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2609.34344 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2609.34344 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co