跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 20 天前AI 評分41

VidDub:面向替換語音的兩步影片配音替換感知蒸餾

VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing

AI 導讀

VidDub 是一款兩步擴散影片配音模型,採用替換感知跨音訊監督,避免來源嘴部動作與替換語音目標衝突,並兼顧唇形同步和視覺保真度。在三個訓練種子中,遮蔽衝突的來源嘴部重建令 cross-audio Sync-C 提升 0.833;最終模型以 2 NFE 在四項基準測試中展現唇形同步、視覺品質、身份保留及視覺語音識別表現。

正文

Published on Nov 2

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

VidDub is a two-step diffusion video dubbing model that uses replacement-aware cross-audio supervision to avoid conflicting source-mouth targets and achieves strong lip synchronization with high visual fidelity.

Video dubbing aims to synthesize lip movements that match target speech while preserving the speaker identity, head pose, and background of a source video. Existing editors are predominantly trained via self-reconstruction under original paired audio, even though the source mouth becomes a conflicting target when driving speech is replaced. We introduce replacement-aware cross-audio supervision: paired-audio samples use full-frame reconstruction, whereas unpaired replacement-audio samples mask out source-mouth reconstruction losses and supervise the mouth solely through audio-visual synchronization while preserving unmasked facial regions and background. Because offline dubbing provides the complete source video and target speech beforehand, we present VidDub, a full-context diffusion editor distilled into two denoising steps (2 NFE). Our replacement-aware teacher first learns accurate lip synchronization without regressing toward conflicting source mouth movements. Matched trajectory regression then transfers the teacher's CFG-guided trajectories to a two-step student, and distribution matching distillation recovers visual quality after step compression. We will release the curated training dataset, VidDub-Bench, fixed paired- and cross-audio evaluation manifests, preprocessing outputs, and unified evaluation code. Across three training seeds, masking out conflicting source-mouth reconstruction improves cross-audio Sync-C by 0.833; the final two-step model combines strong lip synchronization with superior visual quality, identity preservation, and visual speech recognition across four benchmarks.

View arXiv page View PDF Project page GitHub 7 Add to collection

Get this paper in your agent:

hf papers read 2611.03404

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 1

AIPeanutman/VidDub

Video-to-Video •

Updated 1 day ago

•

22

Datasets citing this paper 2

AIPeanutman/VidDub-Training-Data

Viewer •

Updated 1 day ago

•

45.2k

•

208

AIPeanutman/VidDub-Bench

Viewer •

Updated 1 day ago

•

512

•

120

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.03404 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co