在瀏覽器執行 ffmpeg、Whisper 與背景移除：實作經驗

I ship a few "server-class" engines in plain static web pages: ffmpeg.wasm for video, Whisper for transcription, an ONNX segmentation model for background removal, and Tesseract for OCR. No backend at all. They all work, but each one taught me something. Here are the practical lessons.

1. Never load the engine with the page

A 30 MB WASM binary in the critical path will ruin your Largest Contentful Paint and your search rankings. The page should be fully usable (dropzone, options, explanation) before any engine code arrives.

The pattern I use:

// Import only when needed...
async function getFFmpeg() {
  const { FFmpeg } = await import("@ffmpeg/ffmpeg");
  const ff = new FFmpeg();
  await ff.load({ coreURL, wasmURL });
  return ff;
}

// ...but warm the cache once the page is idle
window.addEventListener("load", () => {
  const conn = (navigator as any).connection;
  if (conn?.saveData) return;            // respect Data Saver
  requestIdleCallback(() => fetch(wasmURL));
});

Most users spend a few seconds reading or picking a file. That's enough time to fetch the engine in the background.

2. Show real progress, not a spinner

A spinner during a 40-second Whisper run looks broken. Every engine I use reports progress in some form:

ffmpeg.wasm has a progress event (0–1) and a log stream with time= lines,
transformers.js has progress_callback for model downloads,
tesseract.js has a logger with status and progress.

Show two phases separately: "Downloading model (first time only) 63%" and then "Processing 21%". Users accept a one-time download much more readily when it's labeled.

3. Memory is the real limit

WASM memory is a single linear buffer, and on mobile the tab gets killed long before you'd expect.

Delete files from ffmpeg's virtual FS as soon as you've read the output (ff.deleteFile).
Downscale before ML. Segmentation models work at fixed input sizes anyway, so feeding them a 48 MP photo only wastes memory. Run the model on a smaller copy and apply the mask at full resolution.
Process audio in chunks for Whisper (30 s windows) and don't decode an hour of audio at once.
Release object URLs with URL.revokeObjectURL when you're done.

4. Keep the UI thread free

Anything heavy belongs in a Web Worker. ffmpeg.wasm and transformers.js already run in workers. For my own code (hashing big files, building ZIP/TAR archives) I use fflate's async APIs or a small dedicated worker. A frozen tab during a 2 GB hash makes users think the page has crashed.

5. Cross-origin isolation is a trade-off

Multithreaded WASM needs SharedArrayBuffer, which requires the COOP/COEP headers. Those headers break third-party embeds such as ads and analytics. I use the single-threaded builds wherever the speed difference is acceptable, and keep isolation only on pages that really need it.

6. Cache aggressively

Models and WASM binaries never change for a given version, so serve them with long Cache-Control: immutable lifetimes and versioned URLs. transformers.js also stores models in the Cache API, so the second visit is instant and works offline.

7. Tell users it's local, and let them check

"Your file is never uploaded" is the main selling point, so it should be easy to verify. A one-line note under each tool ("works entirely in your browser") plus the fact that the Network tab stays empty builds more trust than any privacy policy.

You can see these in action, with the Network tab open:

All of them are part of Uttily, a set of free tools that run entirely in the browser. What engines have you got working in the browser? I'm curious what else is possible now.