跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 21 天前AI 評分45

大語言模型智能體能否透過觀看教學影片學習?

Can Large Language Model Agents Learn by Watching Tutorial Videos?

AI 導讀

Tutorial Agentic Learning(TAL)讓大語言模型智能體無須更新參數,便可透過教學影片持續學習。在由 72 部棋類教學影片、涵蓋 12 種開局的 VidLearn 上,TAL 相對原版 MLLM,在訓練及測試集分別取得最高 113% 和 91% 的相對提升。實際對弈中,TAL 在 69% 的正面交鋒局數中取得較佳的開局階段局面。

正文

Published on Nov 1

Authors:

,

Abstract

Tutorial Agentic Learning enables LLM agents to continuously learn from instructional videos without parameter updates by building a structured tutorialbook memory and retrieving via reasoning-linked table-of-contents trees.

Humans continuously acquire knowledge from carefully crafted materials such as books and tutorial videos, carrying that understanding into future tasks. Millions of such instructional videos are uploaded online every day, yet LLM agents remain unable to exploit this ever-expanding wealth of human teaching. Their capabilities are frozen after training, while parameter updates at scale remain prohibitively expensive. This raises the central question: can LLM agents learn from this vast instructional resource without parameter updates? We introduce Tutorial Agentic Learning (TAL), a nonparametric continual-learning paradigm that consolidates tutorial videos into a persistent, structured tutorialbook memory through a two-stage adversarial process that harvests hard, video-grounded questions and uses them to refine the memory. At inference time, TAL retrieves through a Table-of-Contents (TOC) tree, tying retrieval directly to the model's reasoning rather than to embedding similarity. To study this setting, we introduce VidLearn, a multimodal QA benchmark built from 72 chess tutorial videos spanning 12 openings. On VidLearn, TAL outperforms all evaluated baselines and improves monotonically as more videos are consumed, achieving up to 113% and 91% relative gains over the corresponding vanilla MLLMs on the training and test splits, respectively. In actual gameplay, TAL reaches a stronger opening-phase position than its vanilla counterpart in 69% of head-to-head games. Evaluation in a second domain further demonstrates the generality of TAL. Together, these results highlight instructional videos as a scalable resource for parameter-update-free agent continual learning.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2611.01460

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.01460 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.01460 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.01460 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co