跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 3 天前AI 評分59

AURA 提出低延遲因果式串流虛擬形象影片生成方法

AURA: AUdio-dRiven streaming Avatar

AI 導讀

AURA 提出面向串流應用的低延遲因果式音訊驅動影片生成方法,按區塊生成影片,並以 rolling KV cache 和兩階段蒸餾流程減少擴散步數。

正文

Published on Nov 19

Authors:

,

,

,

,

Abstract

AURA enables low-latency, causal video generation for streaming applications by utilizing a rolling KV cache and a two-stage distillation pipeline to significantly reduce the required diffusion steps.

Current audio-driven video diffusion models can synthesize realistic talking-head videos but generally require dozens of diffusion denoising steps over multi-second video chunks, making them unsuitable for streaming applications such as voice assistants and virtual avatars, where low-latency and online causal generation are required properties. AURA generates video chunk by chunk with a rolling KV cache, requiring only the audio of the current chunk to start generating, rather than a multi-second audio look-ahead like bidirectional models. It comes in two sizes: AURA 1.3B streams 576p video in real time on a single GPU (20.7 fps for a 16 Hz playback rate), making it suited to interactive avatar applications, and AURA 5B streams 720p video at 9.7 fps. Our two-stage distillation pipeline first aligns the causal student with the teacher, by training it to match the teacher's trajectories with fewer diffusion steps and a chunk-wise sliding window causal attention mask, then applies Distribution Matching Distillation (DMD) with backward simulation to close the remaining quality gap. On standard talking-head benchmarks, AURA 1.3B runs 23x faster than its own bidirectional teacher and reaches a quality comparable to TalkVerse at a lower resolution, and AURA 5B matches its TalkVerse teacher at 720p while running 17x faster. To support reproducibility and further research on streaming avatar generation, we release the weights of both models along with a dataset of 30K synthetic videos and their precomputed teacher trajectories.

View arXiv page View PDF Project page GitHub 0 Add to collection

Community

Paper submitter

39 minutes ago

Models citing this paper 3

kyutai/AURA-5B

Image-to-Video •

Updated about 2 hours ago

kyutai/AURA-1.3B

Image-to-Video •

Updated about 2 hours ago

kyutai/AURA-1.3B-Teacher-TalkVerse

Image-to-Video •

Updated about 2 hours ago

Datasets citing this paper 1

kyutai/AURA-Dataset

Viewer •

Updated about 2 hours ago

•

30.2k

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.26540 in a Space README.md to link it from this page.

Collections including this paper 1

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co