從瀏覽器串流麥克風音訊至 Gemini Live：打造小型語音助理的實作心得

I wanted to ask my server how it was doing by voice: “what’s the load?”, “how do I free some memory?”. So I built a small FastAPI app that relays microphone audio from a browser to Gemini Live and plays the answer back. The code is open source: https://github.com/yankelperelman/anna-dev. Here is what turned out to matter.

Architecture

Browser ↔ WebSocket ↔ FastAPI ↔ Gemini Live. The server in the middle keeps the API key out of the page and runs the tools (server stats, web search, Telegram) itself. Each connection runs two asyncio tasks, browser→Gemini and Gemini→browser. When either one ends, the other is cancelled.

Capturing audio

The Live API expects 16 kHz 16-bit PCM, while browsers give you 44.1 or 48 kHz floats. I downsample in JavaScript with linear interpolation, convert to int16 and send binary WebSocket frames:

const ratio = inputRate / 16000;
for (let i = 0; i < outLength; i++) {
  const pos = i * ratio, left = Math.floor(pos);
  const right = Math.min(left + 1, input.length - 1);
  output[i] = input[left] * (1 - (pos - left)) + input[right] * (pos - left);
}

This runs in a ScriptProcessorNode, which is deprecated; AudioWorklet is the right tool and is on my to-do list.

Playback

Replies arrive as 24 kHz PCM chunks. I decode each chunk into an AudioBuffer and schedule it right after the previous one:

const startAt = Math.max(ctx.currentTime + 0.03, nextPlayTime);
source.start(startAt);
nextPlayTime = startAt + audioBuffer.duration;

On a good connection this is smooth. On a mobile connection I got stuttering, because late chunks leave gaps. On one carrier it disappeared when I forced 3G instead of 4G. I haven’t found the root cause. A larger lead time trades latency for smoothness, and I experimented with it.

Session lifecycle

The receive iterator finishes at the end of every turn, so the loop has to go around it. An iterator that ends without delivering anything means the session is gone, so I treat that as an error instead of looping forever.

Tool calls

When the model calls a declared function, the server runs it and sends back a FunctionResponse. The system prompt says explicitly never to claim an action was done until the tool returned success. Voice agents are tempted to say “done” first.

Keeping the model on topic

The system prompt limits Anna to server administration and tells her to decline everything else. This is a soft guard: a persistent user can talk around it. If you need a hard limit, remove tools instead of relying on the prompt.

A security detail worth knowing

A local WebSocket server can be reached from any web page open in your browser. I added an Origin check, so other sites can’t connect or change the saved API key, and an optional access-token cookie for servers exposed on a domain.

What’s rough

Deprecated audio API, no anti-aliasing filter in the resampler, no authentication unless you configure a token, and the monitoring skill only works on Linux. Feedback and issues are welcome.