預測在策略自我蒸餾中特權資訊的成效
Predicting Privileged Information Effectiveness for On-Policy Self-Distillation
研究發現,在策略自我蒸餾中,特權資訊的成效取決於內容、生成來源及學生模型;基於 KL 的指標比直接效用估算更能預測訓練收益。研究評估 16 種特權資訊配置、3 個多輪智能體基準及 3 個學生模型,並發現以 Δd^{-}_{err} 為各任務挑選特權資訊,訓練後表現優於隨機選擇。有效的特權資訊配合 GRPO,亦能勝過強力 GRPO 基準。
Published on Oct 15
·
on Oct 20
Authors:
,
,
,
,
,
,
Abstract
On-policy self-distillation with privileged information improves multi-turn agent performance depending on content and source, and KL-based metrics better predict training gains than direct utility estimates.
On-policy self-distillation (OPSD) enables large language models (LLMs) to learn from privileged information (PI) without a separate teacher model. However, prior work largely treats PI as a fixed design choice, leaving the effects of its content and construction poorly understood. This challenge is particularly important in multi-turn agentic settings, where trajectories contain actions, observations, and reasoning that can be transformed into PI in many ways. We systematically evaluate 16 PI configurations across three multi-turn agentic benchmarks and three student models that vary in scale and reasoning mode. We find that PI effectiveness depends on its content, generation source, and the student model. Information-dense trajectories and PI produced by more capable generators generally yield larger gains, whereas thinking models benefit less consistently. To assess PI before training, we compare PI utility with two KL-based metrics: sampled-token reverse KL (D) and error-step KL contrast (Δd^{-}_{err}). The KL-based metrics track post-training performance more consistently than PI utility, which can be distorted by information leakage. Empirically, selecting each task's PI by Δd^{-}_{err} also improves post-training performance over random selection, most when the PI candidates differ widely. Finally, effective PI remains beneficial when combined with GRPO, allowing the combined objective to outperform a strong GRPO baseline. Overall, our results provide practical guidance for constructing and evaluating PI in OPSD training.
View arXiv page View PDF GitHub 1 Add to collection
Get this paper in your agent:
hf papers read 2610.22512
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.22512 in a model README.md to link it from this page.
Datasets citing this paper 1
zkjzou/PI-OPSD
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.22512 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co