AskStream:串流影片理解中何時提問、提問甚麼
AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding
AskStream 是一個研究串流影片理解中何時及應提問甚麼的多輪基準測試,涵蓋影片及用戶的不確定性。基準包含 438 段影片、2,714 個人工標註的影格級時間窗口,分為等待、提問及回答三類,設有四個任務類別和八個子任務。即使最強模型在 Ask-F1 指標上的得分也只有 52.18%,顯示模型在不確定情況下主動尋找資訊仍有侷限。
Published on Oct 28
Authors:
,
,
,
,
,
,
,
,
,
Abstract
Proactive video understanding should include asking for missing information under uncertainty, and the AskStream benchmark reveals current models struggle with this decision.
Recent advances in multimodal large language models have increased interest in interaction paradigms for video understanding. The evolution from offline to streaming and proactive interaction has gradually shifted models from passive responders toward more proactive agents that can decide when to answer. However, existing proactive paradigms typically assume that the information required to answer a query is complete, leaving models to only wait or answer. In the real world, however, information is often incomplete due to uncertainty from either the video or the user. In proactive paradigms, such uncertainty can cause the model to wait forever or generate highly hallucinatory responses. We therefore argue that proactive interaction should be redefined with a third action: ask, allowing models to actively ask for missing information. To study when and what models should ask before answering, we introduce AskStream, the first multi-turn benchmark spanning video and user uncertainty for streaming video understanding, with four task categories and eight subtasks. It contains 438 videos with 2,714 manually annotated frame-level time windows across three types: wait, ask, and answer. Across both multiple-choice and open-ended settings, even the strongest model attains only 52.18% on our proposed Ask-F1 metric, highlighting substantial limitations in proactive information seeking under uncertainty.
View arXiv page View PDF Project page GitHub Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.38678 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.38678 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.38678 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co