跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-15AI 評分38

Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

AI 導讀

Waypoint-OPD 按照教師模型在強化學習早期與後期的策略偏移方向一致性,閘控端點偏移,以改善策略偏移蒸餾。以 RL 訓練的 Qwen3-4B 為教師,在數學及程式推理的六種領域與學生模型設定中,Qwen3-8B、Qwen3-14B 及 Qwen3-30B-A3B 均取得所比較蒸餾方法中最高的領域平均分;相較只傳遞端點偏移,方法只需額外一次凍結教師評分。

正文

Published on Oct 15

·

Submitted by

JGC

on Oct 16

Authors:

,

,

,

,

,

Abstract

Waypoint-OPD improves policy-shift distillation by gating endpoint shifts using directional agreement between early and late teacher policy changes, boosting reasoning performance with minimal overhead.

Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain--student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.21601

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.21601 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.21601 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.21601 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co