跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-06AI 評分41

WorldSonus:為世界模型帶來聲音

WorldSonus: Bringing Sound to Worlds

AI 導讀

WorldSonus 提出面向世界模型的互動式影片轉音訊框架,可即時合成具空間定位的聲音。其串流因果自回歸擴散架構的即時因子(RTF)為 0.41,並以按音訊片段索引的提示詞排程,支援生成期間動態操控聲音事件。在開放領域影片轉音訊基準測試中,WorldSonus 的聲學質素及空間對齊表現匹敵或勝過當前最先進的雙向模型。

正文

Published on Oct 6

·

Submitted by

Zeyue

on Oct 8

Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion. To address these demands, we introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. For real-time generation, WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41. For interactive control, we incorporate an audio-centric captioning pipeline with chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation. For spatial alignment, we leverage high-quality stereo supervision curated from diverse stereo and ambisonic data. Extensive experiments demonstrate that while tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment. Project page: https://noizai.github.io/WorldSonus/

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.08760

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.08760 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.08760 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.08760 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co