跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 18 天前AI 評分40

LiveFoley:為沉浸式互動世界動態即時生成影片音訊

LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds

AI 導讀

LiveFoley 提出串流因果自回歸擴散架構,以極低延遲改善影片轉音訊生成的視聽對齊和音訊質素。Trailing Sliding Window 可將非因果分段編碼器轉為串流模組,而 Unified Varhop Framework 適應不同延遲和運算預算。實驗顯示,LiveFoley 的時間對齊能力媲美離線模型,生成質素領先串流系統。

正文

Published on Nov 4

·

Submitted by

Dai

on Nov 5

Authors:

,

,

,

,

,

,

,

Abstract

LiveFoley improves streaming video-to-audio generation by using a trailing sliding window for visual alignment and expanded temporal denoising windows for audio quality with minimal latency.

Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/

View arXiv page View PDF Project page GitHub Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.05988 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.05988 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.05988 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co