跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-02AI 評分47

On-policy 知識蒸餾(OPD)的效能提升與崩潰:強化學習視角

Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective

AI 導讀

研究從強化學習角度指出,on-policy 知識蒸餾(OPD)可提升語言模型後訓練表現,但亦可能造成過長、重複的生成。OPD 放大教師隱含回饋偏好的學生行為;當偏好與品質不一致時,會出現 reward hacking,但不會擴展學生模型的能力。訓練時屏蔽不健康回應及採用 SFT 初始化,均可有效減輕崩潰。

正文

Published on Oct 2

Authors:

,

,

,

,

Abstract

On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at https://github.com/HancCui/opd_hacking.

View arXiv page View PDF Project page GitHub Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.03185 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.03185 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.03185 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co