當左即是右?CameraVLM 對螢幕以外鏡頭運動的理解
When Left Is Right? Camera-Motion Understanding beyond Screen
CameraVLM 以語言對齊訓練,聯合學習螢幕外鏡頭運動狀態與螢幕內光流動態,改善視覺語言模型對鏡頭運動的理解。這個 9B 參數模型在三項基準及超過 20 個基線上大幅勝過所有開源及鏡頭運動專用模型,並在 48 人研究中媲美或超越 GPT-6 Astra 和人類觀看者。
Published on Oct 12
·
on Oct 15
Authors:
,
,
,
,
Abstract
CameraVLM improves vision-language model understanding of camera motion by jointly learning off-screen motion states and on-screen flow dynamics with language-aligned training.
Can a vision-language model tell left from right? Modern VLMs solve visual puzzles and follow hour-long videos, yet when asked whether the camera moved left or right, most do no better than a coin flip. Adding camera-motion data or geometric representations has been tried, separately, and neither closes the gap. We argue that camera-motion understanding needs two conditions at once: the representation must couple off-screen camera motion with on-screen flow dynamics, and it must be interpreted by the language model rather than trained as a standalone pose or depth estimator. We propose CameraVLM, which pairs a motion-state representation with state-language alignment. Camera tokens capture how the camera moves off screen, flow tokens capture how the subject and background move on screen, and a two-stage training strategy keeps language in the loop so that both are learned together with camera-motion semantics. With only 9B parameters, CameraVLM outperforms all open-source and camera-motion-specific models by a wide margin across three benchmarks and more than 20 baselines, and matches or surpasses GPT-6 Astra as well as human viewers in a 48-participant study. Project page: https://3dagentworld.github.io/CameraVLM/
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.17242
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.17242 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.17242 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.17242 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co