甚麼是即時語音 AI？WebRTC、Whisper 與 LLM 如何協同運作

Have you ever spoken to an AI voice assistant that takes three or four seconds to reply? The awkward pause makes the entire interaction feel unnatural.

In a normal human conversation, the gap between one person stopping and the other starting is tiny. For a machine to replicate this natural flow, a real-time voice AI agent has to hear, think, and speak inside that exact same pause. Hitting this target requires a highly optimized chain of technologies—typically WebRTC, a speech-to-text model like Whisper, a Large Language Model (LLM), and a text-to-speech (TTS) engine.

Here is a breakdown of how real-time voice AI actually works under the hood, and why many engineering teams are moving these pipelines onto dedicated hardware.

The Latency Budget: What Makes AI Feel "Real-Time"?

Speed is everything in voice AI. Industry benchmarks suggest a strict latency budget: your system has about 500 to 800 milliseconds from the end of the caller's speech to the first audio of the reply.

Once delays push past the one-second mark, callers start talking over the AI, assuming it didn't hear them. Hitting a 500ms target means every component must finish its job in fractions of a second:

Inbound Network: 10 to 30 ms
Speech-to-Text (Whisper): 30 to 80 ms
LLM Time to First Token: 150 to 300 ms
Text-to-Speech (TTS): 50 to 100 ms
Outbound Network: 10 to 30 ms

The Cascaded Pipeline: How the AI Thinks

Rather than using one giant end-to-end model, the most practical architecture for self-hosted real-time voice agents is a cascaded streaming pipeline.

WebRTC (The Transport Layer): Callers need a way to send audio without buffering. WebRTC uses the Opus codec, streaming audio instantly (managed by servers like LiveKit).
VAD & Whisper (The Ears): A Voice Activity Detector (VAD) monitors for silence. Once your turn is over, it hands the segment to an optimized model like faster-whisper to transcribe in milliseconds.
The LLM (The Brain): The LLM is the biggest bottleneck. Engineers measure Time to First Token (TTFT) to generate the very first word as quickly as possible.
Text-to-Speech (The Voice): As soon as the LLM finishes its first complete sentence, the TTS engine starts speaking it out loud while the LLM thinks about the second sentence in the background.

Why Run Real-Time Voice AI on Dedicated Hardware?

Running a pipeline this sensitive on shared cloud platforms can be risky. If the platform's scheduling shifts, your latency spikes. Running real-time voice AI on dedicated bare-metal GPU servers solves several major problems:

Predictable, Ultra-Low Latency: Services stay on the same machine, reducing network hops to a fraction of a millisecond.
Direct Network Access: WebRTC requires open UDP ports. A dedicated server gives a direct public IP, bypassing complex NAT layers and load balancers that add delay.
Always-Loaded Models: Large AI models take minutes to load. On a dedicated server, weights remain resident in the VRAM, preventing "cold starts."
Data Privacy: Self-hosting keeps sensitive audio and proprietary logic inside your own environment.

When call volume is highly unpredictable or global distribution is required, elastic cloud scaling still makes sense. But for raw speed and predictable latency, dedicated hardware is becoming the standard.

Building a voice agent is about orchestrating audio, models, and network infrastructure to react at human speed.

Read the full technical deep-dive on our main site here