Fish Audio 即時 STT 每小時 $0.36 並非難點，通話輪替判斷與 Windows 音訊擷取才是工程關鍵

Fish Audio just put a streaming speech-to-text endpoint on the table that claims 2.21% WER on Cekura's Converse-STT leaderboard and conversation-aware turn-taking, priced at $0.36 per session hour. The WebSocket lives at wss://api.fish.audio/v1/asr/realtime. The marketing number is nice. The engineering number that matters for a live interview or call assistant is the gap between the last sample of speech and the moment your UI can show the next suggested reply without talking over the other person.

The turn-taking problem is not a VAD problem

Classic voice-activity detection (energy threshold + hangover timer) works until the speaker takes a breath. On a job interview the silence after "and then we..." is often not the end of the turn. A pure VAD fires, the LLM starts generating, and the candidate is still talking. The reply arrives while they are mid-sentence. Fish Audio's pitch is that the model itself decides when the caller is finished instead of relying on fixed timeouts. That is the right direction. The practical consequence for a desktop app is that you now have two independent clocks:

1. The audio capture clock (WASAPI or Core Audio buffer fills).
2. The STT partial/final clock (WebSocket messages that may arrive 40-200 ms after the audio was sent).

If your overlay or suggestion pane only reacts to final transcripts, you already lost 150-300 ms of usable reaction time. If you react to every partial, you thrash the LLM on incomplete phrases.

What the Windows capture path actually gives you

On Windows the lowest-latency path for a local assistant is still WASAPI exclusive or shared mode with a short period. Shared mode is the realistic choice for an app that must coexist with the call client (Teams, Zoom, Meet).

Key constraints you hit immediately:

• Default shared-mode period is often 10 ms. You can request smaller, but the engine may ignore you.
• Loopback capture (to get the remote party's audio) goes through the same engine and adds another buffer.
• Process-wide audio ducking and communication-device policies can inject extra silence or resampling.

A minimal capture loop that feeds an STT WebSocket looks roughly like this (Python + pyaudiowpatch or a thin WASAPI wrapper):

import pyaudiowpatch as pyaudio
import websocket
import json

p = pyaudio.PyAudio()
# find loopback or mic device
stream = p.open(format=pyaudio.paInt16,
                channels=1,
                rate=16000,
                input=True,
                frames_per_buffer=320,  # 20 ms at 16 kHz
                input_device_index=device_index)

ws = websocket.WebSocket()
ws.connect("wss://api.fish.audio/v1/asr/realtime", header={"Authorization": f"Bearer {user_key}"})

while True:
    data = stream.read(320, exception_on_overflow=False)
    ws.send(data, opcode=websocket.ABNF.OPCODE_BINARY)
    # non-blocking receive for partials
    try:
        msg = ws.recv()
        payload = json.loads(msg)
        if payload.get("is_final"):
            handle_final(payload["text"])
        else:
            update_partial(payload["text"])
    except websocket.WebSocketTimeoutException:
        pass

The 20 ms chunk is a compromise. Smaller chunks increase WebSocket framing overhead and the chance of packetization delay on the far side. Larger chunks hide the true end-of-speech from the model. Most production stacks settle between 10-40 ms.

Latency budget that actually fits in a live call

A usable end-to-end budget for a suggestion that appears before the other person finishes their next sentence:

• Capture + framing: 20-40 ms
• Network + STT first partial: 80-150 ms (varies by provider and region)
• Your local decision logic (debounce, keyterm injection, language detect): 10-30 ms
• LLM time-to-first-token: 150-400 ms depending on model and prompt length
• Render to overlay: <10 ms

Total before you can show something useful: 300-600 ms. That is already tight. Anything that forces a full final transcript before the LLM starts pushes you past the point where the suggestion is still relevant.

The practical mitigations that ship:

• Stream partials into a short rolling buffer and trigger the LLM only on a confidence or silence-from-model signal, not on raw VAD.
• Keep a local draft of the last 1-2 seconds so you can re-prompt with more context when the final arrives.
• Separate the "remote party" track (loopback) from the "local user" track so the STT for the interviewer and the STT for your own speech never contend for the same WebSocket or rate limit.

BYOK and the desktop constraint

Because the key never leaves the machine (DPAPI-protected blob, decrypted only for the outbound Authorization header), you can swap providers the moment one of them raises prices or degrades latency. Fish Audio at $0.36/hour is attractive until the conversation-aware endpoint starts adding 200 ms of extra decision time. Then you fall back to a simpler streaming model that you already have a key for, or to an on-device model if the hardware can take the load. The app itself stays out of the data path.

The same pattern works for any live audio tool: capture locally, send the minimum necessary bytes to the provider the user chose, and keep the decision of when to speak (or when to show a suggestion) under local control rather than under a fixed remote timeout.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.