AURA 2.0:超越回答,學習在串流影片中主動發言與行動
Beyond Answering: Learning to Speak First and Act in Streaming Video
AURA 2.0 是串流影片大語言模型,能在沒有用戶提問時主動開啟對話、調用工具,並進行互動式視覺定位。模型加入 Model-Initiated Conversation(MIC)、Stream-Aware Tool Calling(SATC)、Interactive Visual Grounding(IVG)及長期互動記憶,並採用三階段訓練方案。
Published on Nov 15
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
AURA 2.0 is a streaming video large language model that can initiate conversations, call tools, and perform interactive visual grounding, with long-horizon memory and a three-stage training scheme for improved streaming interaction.
Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) lets the model act on what it sees by invoking external tools using the current visual and conversational context and incorporating returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities.
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2611.17048
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 2
9B • Updated about 5 hours ago • 5 • 13aurateam/AURA_2
9B • Updated about 5 hours ago • 4 • 9aurateam/AURA_2_BF8
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.17048 in a dataset README.md to link it from this page.
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co