Parakeet v3、Whisper 與 Apple 裝置端模型在 25 種語言的比較，公開所有轉錄內容
Parakeet v3 vs Whisper vs Apple's on-device model in 25 languages, every transcript published
Yap 比較 Parakeet v3、Whisper large-v3-turbo 與 Apple macOS 26 裝置端語音辨識模型在 25 種語言的表現，並公開所有轉錄內容。
We make Yap, a dictation app that runs NVIDIA's Parakeet v3 on the device (Mac, Windows and phones). Before saying anything about accuracy in our own marketing, we wanted numbers anyone can check, so we ran three on-device speech recognizers on the same recordings, on one Mac, and published every transcript.
What we compared
- Parakeet TDT 0.6B v3 (NVIDIA), as Core ML through FluidAudio, the same native recognizer our app ships
- Whisper large-v3-turbo (OpenAI), q5_0 quantized, through whisper.cpp with Metal
- Apple SpeechTranscriber, the new on-device model in macOS 26, in the six languages it covers here
All three ran on one Mac with an M2 Pro. Nothing was sent to a server.
Method
The recordings are the FLEURS test split at a pinned revision: 100 recordings per language, taken at evenly spaced positions, in the 25 European languages Parakeet v3 supports. Each engine was given the language. Each transcript and the FLEURS reference sentence were normalized the same way (Unicode NFKC, lowercase, words as runs of letters and digits) and compared word by word. Word error rate is substitutions, deletions and insertions over the number of reference words. Every transcript, timing and the SHA-256 of every recording are in the repository, so every number can be recomputed.
Results: word error rate (lower is better)
| Language | Parakeet v3 | Whisper turbo | Apple (macOS 26) |
|---|---|---|---|
| Bulgarian | 13.4% | 15.9% | not supported |
| Croatian | 12.4% | 11.3% | not supported |
| Czech | 10.4% | 11.5% | not supported |
| Danish | 15.8% | 14.6% | not supported |
| Dutch | 6.3% | 6.6% | not supported |
| English | 4.5% | 3.9% | 6.7% |
| Estonian | 14.2% | 19.9% | not supported |
| Finnish | 10.1% | 7.7% | not supported |
| French | 5.3% | 6.6% | 6.6% |
| German | 4.8% | 5.1% | 6.9% |
| Greek | 31.7% | 12.0% | not supported |
| Hungarian | 14.1% | 17.2% | not supported |
| Italian | 2.5% | 2.7% | 4.2% |
| Latvian | 17.3% | 18.6% | not supported |
| Lithuanian | 17.8% | 25.0% | not supported |
| Maltese | 22.0% | 72.3% | not supported |
| Polish | 8.2% | 7.7% | not supported |
| Portuguese | 3.6% | 3.3% | 5.9% |
| Romanian | 10.0% | 11.8% | not supported |
| Russian | 4.7% | 4.6% | not supported |
| Slovak | 8.4% | 11.5% | not supported |
| Slovenian | 19.8% | 23.5% | not supported |
| Spanish | 2.8% | 2.5% | 4.3% |
| Swedish | 12.0% | 9.9% | not supported |
| Ukrainian | 5.7% | 6.8% | not supported |
The lowest rate in each row is in bold. Parakeet had the lower rate in 15 of 25 languages, Whisper in 10.
What stood out
- Greek is Whisper's by a mile (12.0% against 31.7%). If you dictate in Greek, use Whisper.
- Maltese and the Baltic languages favor Parakeet: Maltese 22.0% against 72.3%, Lithuanian 17.8% against 25.0%, Estonian 14.2% against 19.9%.
- Parakeet beat Apple's new on-device model in all six languages Apple covers here, and Whisper in five: for example German at 4.8% (Parakeet), 5.1% (Whisper) and 6.9% (Apple). In French, Apple edged out Whisper (6.6% against 6.6%, 6.59 to 6.62) while Parakeet led at 5.3%.
- Speed: the median language ran at about 20 times real time with Parakeet and 13 times with Whisper turbo on the same Mac. Apple's model was faster still where it applies.
Limits
- FLEURS is read Wikipedia sentences in quiet rooms. Real dictation, conversation and noise are harder for every engine.
- Numbers and spelling variants are not normalized, so "25" against "twenty-five" counts as an error for all three alike.
- One Mac, one run.
- We make an app that uses Parakeet. That is why everything needed to check us is public.
Data
- Repository with every transcript, timing and recording hash (CC BY 4.0): github.com/founden-bot/on-device-asr-benchmark
- The same data with charts by language: yapapp.cc/data/on-device-asr-benchmark
If you rerun it and get different numbers, or spot a mistake in the scoring, open an issue on the repository.
來源：DEV Community · dev.to