Local Whisper 與雲端聽寫比較：私隱、延遲、準確度取捨及私人設定方法

"Just run Whisper locally" is good advice that skips a lot of detail. Local speech-to-text is genuinely useful, but it is not automatically faster, more accurate, or even fully private, depending on what else sits in your pipeline.

This post breaks the decision into three questions (what leaves the machine, where latency comes from, and how accurate it is for your voice) and ends with a setup where nothing leaves your computer. I use InsertKit as the dictation front end because it supports local Whisper, bring-your-own-key cloud providers, and custom OpenAI-compatible endpoints in one place. The tradeoffs apply to any stack.

1. Privacy: map the data flow first

A modern dictation pipeline usually has two stages:

mic audio -> [speech-to-text] -> raw transcript -> [LLM formatting] -> text at cursor

Each stage can be local or remote, and people often secure one and forget the other.

• Speech-to-text. With a cloud engine, your audio is uploaded and processed by that provider. With a local Whisper model, audio stays on your machine.
• Formatting. Clean-up (removing filler words, fixing punctuation, adapting tone) is usually done by a chat model. If that model is in the cloud, your transcript leaves the machine even when the audio did not.
• Side channels. Sync features count too. In InsertKit, snippets, clipboard history, custom vocabulary, and API keys sync to your account. If your vocabulary list contains an unreleased product codename, that word list is synced, even if every audio clip stays local.

What InsertKit stores is worth knowing precisely: cloud voice history in Settings → Usage covers the last 30 days and is text only, no audio, with a Clear button. Local voice history stays on the device, and setting its maximum to 0 disables it.

So "private" is not a property of a model. It is a property of the whole path.

2. Latency: where the milliseconds actually go

Local is not always faster. Rough breakdown:

Cloud: upload time + queueing + inference on fast hardware + download. On a good connection, short utterances can come back very quickly, because providers run large models on serious GPUs.

Local: zero network, but inference runs on your CPU or GPU. A small model on a modern laptop is responsive. A large model on an older CPU can take noticeably longer than the time you spent talking.

Two details that matter in practice:

• Model loading. Expect the first transcription after loading a large model to be slower than the ones that follow.
• Long dictation. If the tool waits until you finish a three-minute ramble before transcribing, you wait for all of it at the end. InsertKit splits long speech into chunks at natural pauses and transcribes them while you keep talking, which hides much of that delay locally and in the cloud.

3. Accuracy: benchmark your own voice

Published word error rates come from benchmark datasets, not from your accent, your mic, or your jargon. Ten minutes of testing beats any leaderboard.

Here is a quick, tool-agnostic way to compare Whisper model sizes with whisper.cpp:

git clone https://github.com/ggml-org/whisper.cpp
cd whisper.cpp
cmake -B build
cmake --build build -j --config Release

for m in base small medium; do
  sh ./models/download-ggml-model.sh $m
done

# Convert a 60-second recording of you reading real work text to 16 kHz mono WAV
ffmpeg -i sample.m4a -ar 16000 -ac 1 -c:a pcm_s16le sample.wav

for m in base small medium; do
  /usr/bin/time -p ./build/bin/whisper-cli -m models/ggml-$m.bin -f sample.wav -otxt -of out-$m
done

Then score each output against what you actually said:

# pip install jiwer
from jiwer import wer

ref = open("reference.txt").read()
for m in ["base", "small", "medium"]:
    hyp = open(f"out-{m}.txt").read()
    print(m, "WER:", round(wer(ref, hyp), 3))

Use real material: a commit message, a ticket description, a paragraph full of service names. Generic sentences flatter every model.

A few patterns you will likely see:

• Bigger models handle accents and noise better, at the cost of speed and memory.
• Jargon fails across all sizes. That is a vocabulary problem, not a model-size problem.
• Whisper can hallucinate on silence or noise, producing text nobody said. Trimming silence and using a decent mic reduce it.

For jargon, a correction layer helps more than a bigger model. InsertKit's Custom Vocabulary has word hints plus "heard as → write as" corrections, like post gres → Postgres, and corrections run before and after formatting so the LLM cannot undo them.

The decision, simplified

• Audio must never leave the machine: local Whisper.
• Speed on modest hardware: cloud, or a small local model.
• Accuracy on difficult audio: a large local model with a GPU, or a strong cloud engine.
• Zero setup: a hosted engine that needs no key.
• Air-gapped or offline work: local Whisper plus a local LLM, or formatting turned off.

Many developers end up hybrid: local for anything touching client code or internal docs, cloud for low-stakes chat.

A fully private dictation setup

Here is the setup where audio and text both stay on your machine. Local Whisper is a desktop feature in InsertKit (Windows, macOS, Linux).

Step 1: Local speech-to-text

In the Speech Engine panel, choose Local Offline (Whisper) and download a model inside the app. Sizes go from Tiny (~75 MB) and Base (~140 MB) through Small (~460 MB) and Medium (~1.5 GB) to Large v3 (~3 GB). Use your benchmark results to pick. Small is a sensible starting point on most laptops.

Step 2: A local LLM for formatting

Run a local OpenAI-compatible server. With Ollama:

ollama pull llama3.2
curl http://localhost:11434/v1/models

LM Studio exposes a similar server, by default at http://localhost:1234/v1.

In InsertKit, open Settings → API Keys and add a Custom OpenAI-compatible provider: a name, the base URL (for example http://localhost:11434/v1), a key if your server needs one, and the models (auto-fetched or entered by hand). Then choose that model for Smart Formatting.

Smaller local models format well enough for clean-up. If output gets too creative, add a short custom instruction under Use my own instructions, such as "Fix punctuation and remove filler words. Do not add or change content."

Step 3: Decide what happens when formatting is unavailable

Enable Keep working offline so the raw transcript is pasted if the formatter cannot be reached. Or skip formatting for sensitive sessions and rely on vocabulary corrections.

Step 4: Verify, do not assume

Turn off Wi-Fi (or run nmcli networking off on Linux) and dictate a paragraph into your editor. If text still appears and is formatted, both stages are local. Turn the network back on afterwards, because account sync needs it.

Linux note

Check Settings → General → Linux Dependencies. Typing and pasting rely on tools such as xdotool and wtype, and on Wayland the tips point to wtype or ydotool.

Where cloud still wins

To be fair to cloud dictation:

• No local resources. Large models do not eat your RAM or battery during a long build.
• Consistent quality on weak hardware.
• Language coverage. InsertKit's voice typing covers 100+ languages with auto-detect. If you dictate in more than one, benchmark each on your local model before committing to it.
• Mobile. InsertKit's Android app uses hosted or BYOK engines; there is no on-device Whisper there.

If you use the cloud, at least choose deliberately. InsertKit Free needs no key and has a daily limit, Premium uses credits, and BYOK covers providers such as OpenAI, Groq, Deepgram, AssemblyAI, and ElevenLabs.

Wrapping up

Local Whisper is a strong default for private work, but privacy depends on the full path, latency depends on your hardware, and accuracy depends on your own voice. Benchmark with real text, add vocabulary corrections for your stack, and verify your setup with the network off.

If you want to try this exact setup, the desktop builds are at insertkit.com/download.