EvaOmni:以證據為本的視聽推理基準測試
EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning
EvaOmni 是一項以證據為本的基準測試,用於評估全模態語言模型能否識別視聽證據,並據此推理回答問題。基準涵蓋 392 段影片、1,027 組人工標註問答、15 項能力及三個難度層級,並要求模型同時提供答案和支持證據的時間區間。
Published on Oct 20
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
EvaOmni is an evidence-grounded benchmark that evaluates whether omnimodal language models identify audio-visual evidence and construct reasoning chains for audiovisual questions.
Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.
View arXiv page View PDF GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.28444
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.28444 in a model README.md to link it from this page.
Datasets citing this paper 1
VisionXLab/EvaOmni-Dataset
Viewer • Updated 19 minutes ago • 1.03k
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.28444 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co