真實新聞廣播中的七款語音模型：排行榜第一與 0.6B 模型表現相若
Seven speech models on real news broadcasts: Leaderboard #1 ties a 0.6B model
七款語音模型在真實新聞廣播上的 WER 相差不超過 0.94 個百分點，排行榜第一的 30B 模型與 0.6B 模型之間的差異未達統計顯著。測試涵蓋 31 段廣播、共 2.6 小時；速度介乎 13x 至 1,117x，每小時音訊成本約為 $0.003 至 $0.220。選擇模型時，吞吐量和服務配置比模型規模更值得考慮。
The Open ASR Leaderboard has a clear #1: a 30B mixture-of-experts model. Below it sit models a tenth its size.
So, a simple production question: point all of them at real news broadcasts (studio anchors, field reporters, press conferences, background room noise). Does the leaderboard order hold?
It doesn't. On an NVIDIA H100, the seven models land within 0.94 WER points of each other, and a Holm-corrected paired test cannot tell any two apart. What does differ is speed and cost.
The Setup
Wikimedia Commons news (VOA, White House, Al Jazeera)
│ ffmpeg once → 16 kHz mono Opus (byte-identical audio)
▼
human subtitles ──► 15–30 s windows cut at cue boundaries
▼
7 models × same windows · 1× H100 · batch 16 · no language hint
▼
corpus WER · 95% CI by bootstrapping videos · paired test vs best
Bootstrapping videos, not individual windows, matters: twelve windows from one briefing share a room and a microphone. Clustered confidence intervals are wider and honest.
Results Summary
| Model | Released | WER (%) | 95% CI | RTF (Speed) | Cost / Audio Hr |
|---|---|---|---|---|---|
| Parakeet Unified EN 0.6B | 2026-04 | 7.34% | [5.16, 9.97] | 596x | ~$0.005 |
| Qwen3-ASR 1.7B | 2026-06 | 7.58% | [5.53, 9.85] | 89x | ~$0.034 |
| Whisper large-v3-turbo | 2024-10 | 7.64% | [5.10, 10.57] | 115x | ~$0.026 |
| Parakeet TDT 0.6B v3 | 2025-08 | 7.97% | [5.88, 10.41] | 658x | ~$0.005 |
| Nemotron-3.5-Transcribe 30B | 2026-10 | 8.00% | [5.85, 10.46] | 49x | ~$0.061 |
| Granite Speech 5.0 2B | 2026-10 | 8.02% | [5.92, 10.47] | 13x | ~$0.220 |
| freetranscribe parakeet-ft | 2026-10 | 8.28% | [6.14, 10.79] | 1,117x | ~$0.003 |
Statistical verdict: Smallest Holm-corrected $p = 0.654$. At this sample size (31 broadcasts, 2.6 hours), none of the differences between the #1 model and the 0.6B models are statistically significant.
The Dirty Laundry (What Broke)
-
Whisper memorized White House transcripts: On crosstalk, Whisper repeatedly outputted
The Press-. The Press-. The Press-., matching the exact format of the web transcripts it was trained on. It did this across 7 windows; no other model did. - One long window crashed an entire batch: A cue over 30s crashed Whisper's batched call and failed all 16 windows in it (WER spiked to 15.5% instead of 7.6%). When deploying, always retry failed batches item-by-item.
- Reference subtitles vs verbatim transcript bias: Human subtitles edit out stutter words. Verbatim models (Nemotron inserted 628 words, 9% stutter repeats) get billed for faithfulness, while 'cleaner' models matched the human captioner style.
Key Takeaways for Audio Pipelines
- Don't pay for size on English broadcasts. At ~$3/hr for a spot H100, an hour of news audio costs ~$0.003 with the fastest model and ~$0.06 with the slowest, at virtually identical accuracy.
- Select by throughput and serving stack: 0.6B to 1.7B models run comfortably on consumer cards with massive throughput headroom.
Full interactive dot-whisker charts, dataset details, and reproduction methodology:
https://everylocalai.com/benchmark/asr-news-broadcasts-2026-10
來源：DEV Community · dev.to