跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-08AI 評分49

SepGen:單一模型實現多音軌音訊分離及音訊影片生成

SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model

AI 導讀

SepGen 將預訓練音訊影片生成器擴展為可在一次聯合取樣中輸出影片、混合音軌,以及每個文字描述聲源各自的波形。模型設有生成和分離兩種模式:生成模式按聲源描述輸出分軌,分離模式則按文字描述從音訊混合中提取聲音。評測顯示,描述包含對白內容時,SepGen 的表現優於語言條件分離器,尤其在語音分離上;移除對白內容後仍保持領先。

正文

Published on Oct 8

Authors:

,

,

,

,

Abstract

A 4D audio-visual scene comprises a video, the dynamic geometry it depicts, and the sound sources that populate it, each with its own position and trajectory. Rendering such a scene from a novel viewpoint requires that every source be available as an individual waveform, so that it can be localized in the scene and propagated to the observer before the signals are mixed. Joint audio-video generators can synthesize the video and its soundtrack, but the soundtrack is emitted as a single audio-mix in which the sources are not individually accessible. We present SepGen, which extends a pretrained audio-video generator to emit the video, the mixed soundtrack, and one waveform per captioned source in one joint sampling run. SepGen supports two complementary modes: generation and separation. In generation mode, each source caption specifies what its stem contains. A two-speaker dialogue, for example, comes out as one stem per speaker in the original turn order. In separation mode, the input audio-mix remains clean while the captions specify what to extract, so the model can decompose a recording from a free-text description. We evaluate generation on scenes synthesized from text, and separation on scenes rendered by other generators and on real recordings of speech, music, and sound effects. Given an audio-mix and captions that carry the spoken lines, SepGen outperforms language-conditioned separators, most clearly on speech, and it keeps the lead when the lines are removed from the captions. Code, checkpoints, and datasets are available at https://sepgen.github.io/

View arXiv page View PDF Project page GitHub Add to collection

Models citing this paper 2

AviadDahan/SepGen-Separation

Audio-to-Audio •

Updated 3 days ago

•

25

AviadDahan/SepGen-Generation

Text-to-Video •

Updated 3 days ago

•

23

Datasets citing this paper 1

AviadDahan/SepGen-Dataset

Viewer •

Updated 3 days ago

•

4.85k

•

25

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.11361 in a Space README.md to link it from this page.

Collections including this paper 1

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co