跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 13 天前AI 評分36

Reframe:以觀察狀態對齊的多教師在線策略蒸餾,用於智能體視覺推理

Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning

AI 導讀

Reframe 提出兩階段後訓練框架,透過以證據為依據的強化學習及觀察狀態門控的在線策略蒸餾,提升視覺智能體的工具使用能力。框架包含 GD-GRPO 和 OG-MOPD,在 14 個圖像及影片基準測試中均優於 Mix-RL 和 vanilla MOPD。兩個階段均不需要針對任務的 SFT 冷啟動。

正文

Published on Nov 9

Authors:

,

,

Abstract

Reframe is a two-stage post-training framework that aligns visual agents’ observation states through evidence-aware reinforcement learning and observation-gated on-policy distillation, improving tool use and outperforming Mix-RL and vanilla MOPD across image and video benchmarks.

Agentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation. We introduce Reframe, a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment. First, Ground-Drop-GRPO (GD-GRPO) trains spatial and temporal specialists with evidence-aware rewards and stochastic removal of the original visual input, encouraging them to acquire grounded evidence and use the returned observations. Second, Observation-Gated-MOPD (OG-MOPD) replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The two stages complement each other, as learning to rely on tool-returned observations can help the policy benefit from observation repair. Neither stage requires a task-specific SFT cold start. Extensive experiments over 14 image and video benchmarks demonstrate that Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2611.12886

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.12886 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.12886 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.12886 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co