跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-21AI 評分39

RubatoCodec:具固定語音長句事件預算的自適應語音標記化與串流解碼

RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

AI 導讀

RubatoCodec 是基於 X-Codec-2 的可變影格率語音編碼器,以固定的長句事件預算和自適應事件編碼,支援串流解碼。受控實驗顯示,在低事件率下,其重建保真度和清晰度優於固定速率編碼;與其他編碼器相比,它在英語和日語基準測試中保留清晰度的表現最佳。用於文字轉語音時,該表示法取得較低 WER、較高 UTMOS 和 UTMOSv2,説話者相似度則相若。

正文

Published on Oct 21

Authors:

,

,

Abstract

RubatoCodec uses adaptive event-based coding with fixed utterance budgets to improve low-rate speech reconstruction and streaming text-to-speech generation.

Neural speech codecs provide discrete tokens for speech generation. Lower frame rates shorten token sequences, but each token must then carry more linguistic and acoustic information. Variable-frame-rate codecs address this trade-off by adapting token allocation to speech content, but threshold-based allocation lets content determine the token count, so the sequence length a speech generator must produce varies across utterances and datasets at a fixed setting. We present RubatoCodec, a variable-frame-rate speech codec built on X-Codec-2 with an exact utterance-level event budget and streaming decoding. Its Adaptive Firing mechanism uses changes in codec features and waveform spectra to divide speech into variable-duration events without a learned boundary predictor. Discrete codes and an explicit duration represent each event, allowing a speech generator to predict and decode one event at a time. Controlled experiments show that adaptive allocation outperforms fixed-rate coding in reconstruction fidelity and intelligibility at low event rates. A model trained on large-scale multilingual speech achieves competitive reconstruction quality across English, Japanese, and Mandarin, with the best intelligibility preservation among the compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates. In a matched text-to-speech comparison, the adaptive representation yields lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity. The speech generator also learns the codec's utterance-level event budget without an explicit generation-time rate constraint. These results demonstrate that budgeted adaptive speech representations support both high-quality reconstruction and streaming text-to-speech generation. Code and weights are available at https://github.com/third-intelligence/RubatoCodec and https://huggingface.co/third-intelligence/RubatoCodec.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.29434

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 1

third-intelligence/RubatoCodec

Audio-to-Audio •

0.2B •

Updated 7 minutes ago

•

13

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.29434 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.29434 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co