OmniDR-Bench 多模態基準測試揭示 37% 證據差距
Multimodal benchmark exposes 37% evidence gap
OmniDR-Bench 多模態基準測試顯示，智能體即使答題準確，仍有 37% 的視聽證據推理差距。18 組智能體配置中，最佳證據評分僅為 63.3/100；簡報呈現平均 81.8 分，內容及引用質素則分別只有 45.4 分和 47.8 分。
OmniDR‑Bench forces multimodal agents to retrieve, locate, and justify audiovisual evidence, exposing a 37 % shortfall in grounded reasoning despite strong answer accuracy[1]. The benchmark’s three‑step rubric—retrieve the right media, pinpoint the supporting segment, then produce a justification—turns “I think so” into “here’s where it happens”.
Before this work, multimodal evaluation largely measured whether a model could name the correct object or answer a question about a video, using metrics such as VQA accuracy or recall‑based retrieval scores. Those benchmarks never required agents to point to a concrete timestamp or cite a supporting clip, leaving an invisible gap between answering and evidencing.
The best of 18 agent configurations reaches only 63.3 out of 100 on OmniDR‑Bench’s evidence metric[1]. Even the highest‑scoring systems excel at presentation—averaging 81.8 points—but fall to roughly mid‑40s on content (45.4) and citation quality (47.8), showing that polished reports mask weak grounding.
Presentation averages 81.8 points, while content and citation average only 45.4 and 47.8 points respectively[1]. This systematic disparity indicates that current pipelines prioritize surface‑level formatting over the substantive linkage of claims to video moments, a pattern evident in the overall benchmark results.
The benchmark does not yet measure long‑term consistency or cross‑modal causal reasoning beyond the annotated criteria, and the authors note agents still need broader content coverage, stronger causal and logical analysis, and more accurate citation support[1]. This suggests that improving raw retrieval or language modeling alone will not close the gap; dedicated grounding architectures and multimodal reasoning components are required.
Future model releases should treat the evidence score as a primary KPI, prompting developers to integrate segment‑level grounding modules before publishing answer‑only APIs. The clear 37 % shortfall sets an immediate target: raise content and citation performance into the 70s, or risk lagging behind real‑world research assistants that must back every claim with concrete audiovisual proof.
References
來源：DEV Community · dev.to