Talvio 開發者分享以 Tauri 和 Rust 開發開源聽寫應用時遇到的 5 個問題

I type a lot less than I used to. I hold two keys, say what I want, let go, and the text shows up wherever my cursor is. Slack, VS Code, a Gmail draft, a terminal. It works with the Wi-Fi off.

That app is Talvio. It's open source (MIT), built with Tauri 2, Rust and React, and it runs on Windows and Linux today. A macOS build is coming.

GitHub: github.com/ishangautam7/Talvio

This post is half "here's what I made" and half "here's what broke". If you build desktop apps with Tauri, the second half is probably the useful part.

Why I built it

I wanted dictation that worked in every app, not just in one website's text box. I also didn't want my voice going to a server every time I wrote a Slack message.

I found Handy by CJ Pais. It's a small, open source, offline speech-to-text app, and it did the core thing well: hold a key, talk, paste. I forked it. The whole local dictation core (the Whisper/Parakeet pipeline, voice activity detection, the recording overlay, the settings architecture) comes from that project, and I still pull fixes from upstream. Go give it a star too.

Then I kept adding the things I wanted on my own machine. That fork grew into Talvio.

What it does

Dictation. Hold Ctrl + Win on Windows (Ctrl + Space on Linux), talk, let go. A small pill at the bottom of the screen shows it's listening, and with a streaming model you see your words as you speak.

Speech recognition runs locally through transcribe.cpp. Whisper, Parakeet, Nemotron and the rest of a 69-model catalog, on your GPU through Vulkan or on the CPU. No account needed. Cloud speech providers are optional and use your own key.

An assistant on another shortcut. Select some text, hold Ctrl + Alt, say "write a polite reply saying I can't make Thursday". It answers out loud, and an Insert button puts the answer back where the text was.

Voice commands. "Play Believer on YouTube." "Turn the volume down." "Email Ram that I'm late." These run in your own apps: Gmail opens with the email written, the song starts in your browser.

Meeting notes. It records your mic and the system audio as two separate lanes, so it knows which words were yours. No bot joins the call. You get notes at the end.

Smaller things. Optional AI cleanup that removes "um" and follows "at ten, actually no, eleven". A dictionary so names come out spelled right. And an Android app that turns your phone into a wireless mic, keyboard and remote for the PC.

The AI features (assistant, commands, cleanup) use your own API key, a local model through the built-in llama.cpp engine, or $1 of free credit if you sign in. Plain dictation needs none of that.

OK, the lessons.

1. Creating a window from a hotkey thread froze the whole app

The assistant panel opens when you press a global shortcut. The shortcut callback runs on the keyboard hook's thread, not on the main thread.

My first version built the WebView window right there in the callback. It worked most of the time. Sometimes it deadlocked the event loop, and then everything was dead: no overlay, no dictation, windows you couldn't drag.

The fix is boring. Every function that creates, shows or destroys that window queues the work on the main thread:

pub fn enter_conversation_size(app: &AppHandle) {
    let app_main = app.clone();
    if let Err(e) = app.run_on_main_thread(move || {
        apply_panel_geometry(&app_main);
        place_call_window(&app_main);
    }) {
        error!("Could not queue conversation panel sizing: {}", e);
    }
}

The functions that actually touch the window are private. The public ones only queue. That had a side benefit I didn't plan for: a create is always ordered behind a destroy, so toggling the feature off and on fast can't leave the app thinking a window exists when it doesn't.

If you call anything window-related from a shortcut handler, a tray handler or a background thread in Tauri, go through run_on_main_thread.

2. Floating windows must never take focus

A dictation app pastes with a synthetic Ctrl+V. That keystroke goes to whatever has keyboard focus at that moment. If your own overlay grabbed focus, the transcript gets pasted into your overlay and disappears.

This happened. The overlay window was bigger than the visible pill (a transparent frame around it), so a click on "empty" space actually landed on my window, activated it, and the next paste went nowhere.

Each platform needs a different answer:

• Windows: the window gets WS_EX_NOACTIVATE, so even a click that lands on it can't make it the foreground window.
• macOS: an NSPanel that never becomes key.
• Linux (Wayland): a GTK layer-shell surface with KeyboardMode::None.
• Plain X11: none of the above works reliably, so the card there stays click-through until the paste is done.

The same rule now applies to every window that pops up on its own: the reminder popup, the "allow this phone?" card, the assistant. Anything that shows up while you are typing somewhere else should not move your cursor.

One more trap: keeping it on top. WS_EX_TOPMOST is a position another app can take from you. I set it once on show and users reported the pill "randomly hiding". Now a small guard re-asserts it every 700 ms while the overlay is up, and logs once when something displaced it. That log line turned a vague bug report into a fixable one.

3. Run the speech engine in a separate process

Local speech recognition on the GPU means running Vulkan compute shaders through whatever driver the user has. Some of those drivers crash. Some hit a GGML_ASSERT and abort. Some just hang.

When the engine ran inside the app, any of those took the whole app down, or left it sitting on "Transcribing…" forever.

Now every call into the speech engine goes to a worker process: the same executable started with --transcribe-worker. main.rs checks for that flag before anything else (before Tauri, logging or single-instance), so the worker is just a small loop reading requests from stdin.

The protocol is plain request/response, one frame per message:

[u32 json length][json][u32 pcm length][f32 pcm samples]

Ten minutes of audio costs about 10 ms to send this way. There's a 1 GiB cap per section so a garbage length can't turn into a huge allocation.

When a worker using the GPU crashes or hangs, the app marks the GPU unavailable for the session and retries once in a worker started with --cpu-only. That flag restricts the engine to CPU backends before any GPU code runs, so a driver that dies during init can't take the fallback down too. The user gets their text, a bit slower, and a log line saying why.

Two details that matter more than they look:

• On Windows the worker goes into a kill-on-close Job Object, and it exits when its stdin closes. A crashed app never leaves a worker eating your GPU memory.
• Esc goes through the engine's abort callback first and only kills the worker after 750 ms. Killing it every time meant reloading the model after every cancelled dictation.

4. My settings file got wiped on shutdown

Early on I used tauri-plugin-store for settings. Then a shutdown reset every setting. Shortcuts, models, everything back to defaults.

Two things combined. The plugin wrote the file in place, so a shutdown at the wrong moment (or Windows Defender holding the file) could leave it empty or half-written. And on the next launch, a file that didn't parse was silently treated as "no settings", so defaults were written over it.

The replacement is small:

pub fn write_atomically(path: &Path, bytes: &[u8]) -> io::Result<()> {
    // write to a temp file next to the real one
    let mut file = retry_while_held(|| File::create(&temp))?;
    file.write_all(bytes)?;
    file.sync_all()?;
    drop(file);
    // then swap it in with one rename
    retry_while_held(|| fs::rename(&temp, path))?;
    Ok(())
}

retry_while_held retries for a few hundred milliseconds, because on Windows an antivirus, the search indexer or a sync client often has the file open for a moment.

The rest is about what happens when the file is damaged anyway. A broken file is moved aside, not overwritten. Settings come back from a .bak copy that is only refreshed from a launch that read a good file. A read error (as opposed to bad content) never triggers a reset at all. And the user gets a notice instead of silently losing their setup.

If you store anything users care about, don't let "can't parse it" mean "start fresh".

5. You don't need paid code signing to ship auto-updates

I assumed self-update was tied to Apple and Windows code signing certificates. It isn't. Tauri's updater checks a minisign signature made with a key you generate yourself, for free. Code signing is a separate thing that only affects the "unknown publisher" warning on first install.

I had tied them together in CI. The result: from 1.0 to 1.4, no release had a working update path, and nobody noticed because the update UI was also hidden. Fun.

What the pipeline does now:

• Build with the updater key from repo secrets whenever it exists, signed or not.
• Re-sign the Linux AppImage and .deb after they are repacked. The bundler's signatures describe the bytes before repacking.
• Write latest.json once, at the end, from the signatures actually attached to the release, and verify each one against the published file before it goes live.

The app polls releases/latest/download/latest.json. GitHub only serves that from the newest published, non-prerelease release, so drafts and betas never reach anyone.

And back up that private key. If you lose it, installed copies can never update again.

Try it

• Download: GitHub releases (v2.2.0) or talvio.sentramind.dev
• Windows installer isn't code signed yet, so SmartScreen will ask. More info, then Run anyway.
• Linux: AppImage or .deb.

It's a solo project and there are rough edges. If something breaks on your machine, an issue with your GPU and OS helps a lot. If it's useful to you, a star on GitHub helps more people find it.

Repo: github.com/ishangautam7/Talvio

I'd also like to know: which of these did you hit in your own Tauri or Electron app? The focus one took me embarrassingly long to understand.