WaveGen:透過內部頻譜軌跡學習實現真正端到端的波形生成
WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning
WaveGen 提出以 Spectral Forcing 為基礎的端到端擴散式文字轉波形框架,在單一模型內共同學習頻譜與波形軌跡,無需外部音訊編解碼器、聲碼器、潛在模型或對齊模組。在 Seed-en benchmark 上,WaveGen 的 WER 為 1.47、SIM 為 0.75,並在 Emilia 訓練數據集上達到最先進的文字轉語音表現;模型規模由 0.1B 擴展至 0.6B 參數。
Published on Nov 9
Authors:
Abstract
WaveGen is a fully end-to-end diffusion framework for text-to-waveform generation that jointly learns spectral and waveform trajectories via Spectral Forcing without external audio codecs, vocoders, latent models, alignment modules, or self-supervised representations, achieving state-of-the-art text-to-speech results.
This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. In addition, WaveGen performs internal text-speech alignment within a single model, eliminating the need for external alignment modules such as duration predictors. To preserve a truly end-to-end formulation, our framework further avoids dependence on self-supervised representations, such as BERT-based text embeddings and Wav2Vec 2.0-based semantic representations, as well as semantic distillation methods for training optimization. To achieve this, we carefully design Spectral Forcing, a model architecture with disentangled diffusion heads, and a training objective based on a multi-scale spectral v-loss. We further validate the scalability of end-to-end TTS models by scaling the model size from 0.1B to 0.6B parameters. Using a single architecture, open-source data, and one-stage training, without external modules such as duration predictor or REPA, WaveGen demonstrates fully end-to-end waveform generation achieving state-of-the-art performance on the Emilia training dataset. In particular, WaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark. Source code and checkpoints are available at https://github.com/sh-lee-prml/WaveGen
View arXiv page View PDF Project page GitHub 11 Add to collection
Get this paper in your agent:
hf papers read 2611.12527
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 1
Text-to-Speech • Updated about 21 hours agoLeeSangHoon/WaveGen
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.12527 in a dataset README.md to link it from this page.
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co