跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-08AI 評分38

ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills

ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills

AI 導讀

ViSkill 以視覺技能卡記錄成功互動,讓 VLM 智能體在推理和獎勵塑形時使用技能,並將成功軌跡回饋至技能庫,形成閉環學習。ViSkill 在 Sokoban、FrozenLake 和 PrimitiveSkill 上的整體成功率為 0.89,加入冷啟動初始化後升至 0.91,並勝過所有受評估的專有及開源基準模型,收斂速度亦快於標準 PPO。

正文

Published on Oct 8

Authors:

,

,

,

,

,

,

,

Abstract

Skill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. Yet most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language that loses critical geometric structure. Recent efforts have begun incorporating visual evidence, but construct and update skills separately from policy optimization, leaving their mutual improvement underexplored. We propose ViSkill, a visual-native skill learning framework that encodes successful interactions as composite visual skill cards directly accessible to VLM agents. Retrieved skills guide both inference and reward shaping, while successful trajectories are distilled back into the library, forming a closed feedback loop in which skill accumulation and policy improvement reinforce each other. An optional cold-start mechanism further accelerates early-stage learning. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, ViSkill achieves an overall success rate of 0.89, rising to 0.91 with cold-start initialization, outperforming all evaluated proprietary and open-source baselines while converging faster than standard PPO. Our code is available at https://github.com/ZJU-REAL/ViSkill.

View arXiv page View PDF GitHub Add to collection

Models citing this paper 2

hongxingli/ViSkill-Sokoban

Image-Text-to-Text •

4B •

Updated 15 minutes ago

hongxingli/ViSkill-FrozenLake

Image-Text-to-Text •

4B •

Updated 15 minutes ago

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.12403 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.12403 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co