跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-23AI 評分50

Falcon:面向真實聲學場景的高效串流音訊視訊目標説話者提取

Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

AI 導讀

Falcon 是一個因果式音訊視訊目標説話者提取模型,利用目標説話者的嘴部影片,從嘈雜及重疊的語音中提取其聲音。論文提出 RealSSA 基準,涵蓋不同説話者數量、遠近場幹擾、混響及突發噪音,並包括模擬場景和保留的真實錄音評測集。

正文

Falcon is a causal audio-visual target speaker extraction model that uses the target speaker's mouth video to extract their voice from noisy, overlapping speech. It combines a single-pass multi-scale time-frequency separator with an encoder-space speaker contrastive loss that reduces interfering-speaker leakage without adding inference cost.

The paper also introduces RealSSA, a benchmark covering varied speaker counts, near- and far-field interference, reverberation, and sudden noise, with both simulated scenes and a held-out real-recorded evaluation set. Compared with causal AV-TFGridNet, Falcon reduces trainable parameters by 91.9%, computation by 20.2×, and GPU inference latency by 11.1× in the reported evaluation. Code, model weights, datasets, and interactive demos are available on the project page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co