跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-14AI 評分37

Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

AI 導讀

CriticOPD 透過批判引導的 On-Policy Distillation,改善 OPD 的監督可靠性:教師模型會參考標準答案批判學生的錯誤推理,學生再根據回饋改寫後續內容。六個數學推理基準測試中,CriticOPD 的平均準確率較 OPD 在 Qwen3-1.7B 提升 4.57 個百分點,在 Qwen3-0.6B 提升 4.06 個百分點;競賽題目上的提升最大。

正文

🤖 CriticOPD: Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

🔍 Key finding: Teachers can recognize student mistakes yet still follow them when continuing a flawed solution, making OPD supervision unreliable.

✍️ Our approach: The teacher uses the reference answer to critique incorrect student rollouts, and the student rewrites the continuation with this feedback. The teacher scores the rewritten suffix with the feedback in context, while the student learns from that suffix without seeing the feedback.

📈 Results: Across six math reasoning benchmarks, CriticOPD improves average accuracy over OPD by 4.57 percentage points on Qwen3-1.7B and 4.06 on Qwen3-0.6B, with the largest gains on competition problems.

🌐 Project page · 💻 Code

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co