WorldSonus:為世界模型帶來聲音
WorldSonus: Bringing Sound to Worlds
WorldSonus 提出面向世界模型的互動式影片轉音訊框架,可即時合成具空間定位的聲音。其串流因果自回歸擴散架構的即時因子(RTF)為 0.41,並以按音訊片段索引的提示詞排程,支援生成期間動態操控聲音事件。在開放領域影片轉音訊基準測試中,WorldSonus 的聲學質素及空間對齊表現匹敵或勝過當前最先進的雙向模型。
Published on Oct 6
·
on Oct 8
Authors:
,
,
,
,
,
,
,
,
,
,
Abstract
Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion. To address these demands, we introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41. For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation. For spatial alignment, we leverage high-quality stereo supervision curated from diverse stereo and ambisonic data. Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment. Project page: https://noizai.github.io/WorldSonus/
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.08760
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.08760 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.08760 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.08760 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co