跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-07AI 評分55

On-Policy Distillation 能教會模型新技能,卻不傳授新知識

On-Policy Distillation Teaches New Skills but Not New Knowledge

AI 導讀

這項研究發現,reverse-KL on-policy distillation(OPD)能把組合推理技能遷移至未見過的推理結構,但只轉移極少事實知識。在四個、來自三個模型家族的模型中,以 forward KL 取代 reverse KL 可恢復事實遷移,而學生模型 rollout 則提升多步推理的執行能力。

正文

Published on Oct 7

·

Submitted by

yixuan

on Oct 9

Authors:

Abstract

On-policy distillation (OPD) strengthens language-model reasoning, yet whether students acquire new factual knowledge or compositional skill for multi-step reasoning remains unknown. We separate these capabilities using a controlled synthetic framework that measures the student's initial capabilities and independently controls the teacher's additional facts, compositional skill, or both. Across four models from three families, reverse-KL OPD reliably transfers compositional skill across unseen reasoning structures, but transfers minimal factual knowledge. Decoupling the distillation recipe reveals the source of this asymmetry: replacing reverse KL with forward KL restores factual transfer, whereas student rollouts specifically improve the execution of multi-step reasoning. Experiments on recent factual QA and competition mathematics show a similar asymmetry under reverse-KL OPD, yielding notable reasoning gains without factual memory expansion. Together, these results demonstrate that on-policy distillation does not expand a model's parametric knowledge, but instead teaches it to organize and compose the knowledge it already possesses.

View arXiv page View PDF Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.09639 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.09639 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.09639 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co