Video-Arena:面向時空能力的可擴展智能體影片基準測試
Video-Arena: A Scalable Agentic Video Benchmark for Spatiotemporal Capability
Video-Arena 將競技遊戲轉化為可擴展、可驗證的影片基準,評估模型理解及追蹤動態世界狀態的時空能力。基準集包含涵蓋 9 款遊戲的 180 項任務,並以原生視覺理解及工具輔助智能體分析兩種評估協定進行測試。結果顯示,原生模型難以可靠感知事件;工具輔助系統雖能改善事件還原,仍難以持續解讀狀態更新及追蹤結果狀態。
Published on Oct 9
Authors:
,
,
,
,
,
,
,
,
Abstract
Understanding and tracking a dynamic world are fundamental to visual intelligence. Existing video benchmarks provide limited insight into whether models can maintain a coherent understanding of evolving world state. Building such a benchmark from real-world videos is challenging because the underlying world state is partially observable, making annotations hard to obtain. To close this gap, we introduce Video-Arena, a benchmark that turns competitive games into scalable, verifiable tests of how well models understand evolving world states from videos. Specifically, inspired by how humans understand dynamic events, we evaluate spatiotemporal world state understanding and tracking through three task categories: perceiving critical events, interpreting state updates, and tracking the resulting states. Our pipeline can generate videos and aligned reference labels from game records at scale. To keep evaluation affordable and reproducible, we curate a golden set of 180 tasks across 9 games and run evaluations under two protocols: native visual understanding and tool-assisted agentic analysis. The best-performing systems achieve F1 scores of and under the two protocols, respectively. The result highlights that native models struggle to perceive events reliably, while tool-assisted systems improve event recovery but still struggle to interpret state updates and track the resulting states consistently over time. Our project page is in https://videoarena-bench.github.io
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.14158
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.14158 in a model README.md to link it from this page.
Datasets citing this paper 1
THUSI-Lab/Video-Arena
Updated about 2 hours ago • 75
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.14158 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co