OpenMOSS/TTSD-eval:多説話人對話生成的客觀評測流程
OpenMOSS/TTSD-eval
OpenMOSS/TTSD-eval 提供多説話人對話生成的客觀評測流程,以説話人歸屬準確率(ACC)和説話人相似度(SIM)為核心指標。WER 作為評估可理解度的補充指標,數值愈低愈好;測試集設有中文和英文子集,每種語言各含 50 段對話樣本。流程使用 MMS-FA 對齊輸入文字與生成音訊,並以輸入稿的説話人標籤核對預測結果。
This repository provides an objective evaluation pipeline for multi-speaker dialogue generation. The core metrics are:
- Speaker attribution accuracy (ACC).
- Speaker similarity (SIM).
As a supplementary metric, we also report Word Error Rate (WER, lower is better) to assess intelligibility.
Evaluation Framework
- Traditional dialogue evaluation is often constrained by speaker diarization performance.
- This project uses forced alignment to build a unified and more stable evaluation pipeline.
- We first apply MMS-FA to align source text and generated audio at word level.
- Audio is segmented into sentence fragments by punctuation, and speaker labels are assigned from explicit speaker tags in the input script.
- We use
wespeaker-SimAMResNet100as the speaker embedding model. - For each fragment, we compute similarity against prompt audio from both speakers; the higher score determines the predicted speaker.
- Comparing predicted labels with script labels yields ACC, and similarity scores yield SIM.
Test Set
- The benchmark contains Chinese and English subsets, with 50 dialogue samples per language.
- For each language, 20 speaker-reference pairs are manually collected from public Internet sources, and 30 pairs are sampled from
seed-tts-eval. - Dialogue texts are generated with Gemini 2.5 Pro.
- Audio durations span roughly 30s to 720s, covering both short dialogues and long-form multi-turn interactions.
Unzip test set
git lfs install && \ git lfs pull && \ unzip -oq testset.zip \ -d .
After extraction, files are located in:
testset
Installation
Install dependencies with conda and pip.
Using conda
conda create -n ttsd_eval python=3.12 -y && conda activate ttsd_eval
pip install -r requirements.txtModel Preparation
1. Download WeSpeaker Weights (Required)
The default model directory used by tools/run_similarity.py is:
model/voxblink2_samresnet100_ft
Download the model package:
https://wenet.org.cn/downloads?models=wespeaker&version=voxblink2_samresnet100_ft.zip
Unzip to model/:
mkdir -p model && \
unzip -oq model/voxblink2_samresnet100_ft.zip -d model2. Pre-download Alignment Model (Recommended)
mkdir -p model/checkpoints && \ wget -c -O model/checkpoints/model.pt "https://dl.fbaipublicfiles.com/mms/torchaudio/ctc_alignment_mling_uroman/model.pt"
Usage
1. Similarity Pipeline (eval.sh)
eval.sh runs:
tools/align.pytools/split.pytools/run_similarity.py
Before running, set INPUT_JSONL in eval.sh to your input JSONL paths.
bash eval.sh
Output files are named as:
<input_file_stem>_<timestamp>_with_alignment.jsonl<input_file_stem>_<timestamp>_with_alignment_split.jsonl<input_file_stem>_<timestamp>_with_alignment_split_with_sim.jsonl
2. WER Pipeline (run_wer.sh, Optional)
run_wer.sh runs:
wer/whisper_asr.pywer/run_wer.py- WER is treated as a supplementary metric for intelligibility (lower is better).
- ASR uses Whisper-large-v3 by default.
- Before WER computation, speaker/sound-event tags are stripped, and both reference text and ASR output are normalized in
seed-tts-evalstyle.
Before running, set input_jsonl_list in run_wer.sh.
bash run_wer.sh
Output files are named as:
<input_file_stem>_<timestamp>.jsonl<input_file_stem>_<timestamp>_with_wer.jsonl<input_file_stem>_<timestamp>_with_wer.txt
Input JSONL Format
Each line in the input file must be a valid JSON object (JSONL format).
Required fields for eval.sh
text: dialogue script with explicit speaker tags such as[S1]and[S2].output_audio: generated dialogue audio path.prompt_audio_speaker1: reference audio for speaker 1.prompt_audio_speaker2: reference audio for speaker 2.
來源:OpenMOSS / 復旦NLP / 上海創智 / MOSI · github.com
