PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
PR-OPD 是一種面向智能體強化學習的在策略自我蒸餾方法,透過對齊策略自身回應的隱藏狀態來利用技能資訊。在 ALFWorld 和 WebShop 的所有測試設定中,PR-OPD 均取得最佳整體結果,較 GRPO 分別最多提升 4.7 個百分點和 14.0 個百分點。此方法以 GRPO 預熱,無需外部技能庫或獨立教師,也沒有推理開銷。
Published on Sep 29
Authors:
,
,
,
,
,
,
Abstract
Language-model agents are usually trained by reinforcement learning from one reward per episode, and privileged self-distillation enriches it by letting the same policy, given a skill, teach its skill-free self through token probabilities. However, we identify two phenomena that question this channel. Invisible Advantage: a skill in context lifts WebShop success from 42.2% to 56.2%, yet changes the probabilities of fewer than a quarter of the sampled tokens. Much to Align: a skill changes the hidden states of over 80% of response tokens, in a way that linear probes can trace back to the specific skill. To exploit this, we propose Privileged Representation On-policy Self-Distillation (PR-OPD). After a GRPO warm start, the policy writes a hindsight skill for each trajectory, re-reads its own responses with that skill as a stop-gradient teacher, and aligns its projected hidden states to the teacher's at every layer alongside the reward objective, with no external skill library, separate teacher, or inference overhead. On ALFWorld and WebShop with two backbones, PR-OPD achieves the best overall results in every setting, improving over GRPO by up to 4.7 points in ALFWorld success and 14.0 points in WebShop accuracy. Code is available at https://github.com/balibata/PR-OPD.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2609.36642
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.36642 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.36642 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.36642 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co