SleuthVLM:全模態影片理解真的需要影片音訊嗎?
SleuthVLM: Do We Really Need Audio-in-Video for Omni Video Understanding?
SleuthVLM 論文提出一種不直接聽取音訊的影片理解智能體,透過語音轉錄及自主搜尋影片內容,在四個全模態基準上取得 75.0% 平均準確率。影片最長達 90 分鐘;該成績較 Gemini 3.8 Flash 高 2.6 個百分點,並以 77.8% 登上 Video-MME-v2 排行榜首。
Published on Nov 4
Authors:
,
,
,
,
,
,
,
Abstract
SleuthVLM uses a vision-language model with transcribed speech and autonomous video sleuthing to outperform direct audio-hearing omni models on long-video benchmarks, showing that agentic reasoning matters more than auditory input.
Omni models answer questions about videos by jointly processing sampled frames and audio in one turn, with progress driven largely by scaling model size and omni training data. We ask whether a model needs to hear the audio directly at all, and present SleuthVLM, an agent harness built on a VLM that never hears the audio directly: speech will be transcribed into text, and the VLM will sleuth the video on its own without any predefined workflow. On four omni benchmarks with videos up to 90 minutes, SleuthVLM achieves the best average accuracy: 75.0%, 2.6 points above Gemini 3.8 Flash, and it also tops the Video-MME-v2 leaderboard with 77.8%; with an open-source 27B VLM, it beats all open-source omni models. We draw three findings: SleuthVLM beats a model that glances; hearing is not the bottleneck, agentic ability is; omni understanding needs better agentic ability, not just more omni data for modality alignment. So we provide a simple but effective starting point for doing omni video understanding. Omni understanding needs better agents, not just bigger models that hear.
View arXiv page View PDF Project page Add to collection
Get this paper in your agent:
hf papers read 2611.06083
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.06083 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.06083 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.06083 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co