跳到正文
DEV Community· 王果ai·· 6 天前AI 評分55

WebSockets 為何會令電話語音 AI 失靈:以原始 8kHz RTP 建立 60ms 插話迴路

Why WebSockets Break Telephony Voice AI: Building a 60ms Barge-In Loop on Raw 8kHz RTP

AI 導讀

作者展示以原始 8kHz RTP 建立的電話語音 AI 插話架構,透過約 50–60ms 的快速迴路觸發外播音訊緩衝區清空。架構按 20ms RTP 影格處理音訊,以 SOXR VHQ 將 8kHz 重採樣至 16kHz,並在 0.16ms 內清空 256-Frame 外播環形緩衝區。

正文

王果ai


1. The Telephony Gap: Browser Demos vs. PSTN Reality

Most Voice AI tutorials work great over a clean laptop microphone in a quiet room using browser WebSockets. The moment you connect that same pipeline to real enterprise telephony (PSTN / SIP trunks / FreeSWITCH), three brutal physical realities hit at once:

  1. The 8kHz vs. 16kHz Sample-Rate Wall: Traditional PSTN/SIP networks stream 8kHz G.711 (PCMU/PCMA) or L16 audio, while modern acoustic models (SenseVoice, Whisper, Silero VAD) expect 16kHz PCM. Naïve linear interpolation introduces aliasing artifacts that tank ASR accuracy and cause false VAD triggers.
  2. TCP Head-of-Line Blocking: Tunneling continuous 20ms telephony audio frames over TCP WebSockets creates packet-queue stalls under cellular jitter—turning a 50ms interruption into a 700ms cross-talk collision.
  3. Echo & Unflushed Outbound Ring Buffers: Even when your backend decides to stop speaking, an unflushed 20ms–200ms outbound RTP playback queue keeps talking over the caller.

2. The 60ms Enterprise RTP Barge-In Architecture

In our open-source lab (01-voice-agent-rtp-bargein), we bypassed generic JSON-over-WebSocket wrappers and engineered a deterministic telephony loop:

[PSTN / SIP Caller (8kHz RTP, 20ms frames)]
        │
        ▼ (UDP Raw RTP)
[FreeSWITCH Media Gateway]
        │
        ├──► [SOXR VHQ Upsampler (8kHz ➔ 16kHz)] ──► [SenseVoice / Acoustic VAD (~18ms)]
        │                                                       │
        │                                                       ▼ (Barge-In Triggered!)
        └──◄ [256-Frame Outbound Ring Buffer] ◄──── [0.16ms clearAudio() Hard Flush]

Key Architectural Primitives

  1. 20ms Raw RTP Frame Alignment: Audio arrives in strict 20ms packets (160 samples at 8kHz). Processing directly on frame boundaries avoids arbitrary chunk buffering delays.
  2. SOXR VHQ Zero-Artifact Resampling: Using SoX Resampler (SOXR_VHQ) for 8kHz <-> 16kHz conversion preserves vocal formants and prevents high-frequency aliasing from poisoning the local VAD classifier.
  3. 256-Frame Outbound Ring Buffer + 0.16ms clearAudio(): Rather than letting TTS audio accumulate in an uncontrolled OS socket buffer, outbound TTS frames sit in a fixed 256-frame circular buffer. The instant acoustic barge-in fires, clearAudio() resets the read/write pointers in 0.16ms—producing immediate silence on the caller's phone.

3. Why Decoupling Interruption from Transcription Matters

A common mistake in Voice AI is waiting for the LLM or full-sentence STT transcript to confirm "the user is interrupting". Transcription requires context windows (often 200ms–400ms).

By splitting the pipeline into two asymmetric loops:

  • Fast Loop (~50–60ms): Acoustic VAD + energy/formant gate ➔ triggers immediate clearAudio() buffer flush.
  • Semantic Loop (~200–400ms): Full ASR transcript ➔ feeds the LLM context & updates turn state (discarding false backchannels like "uh-huh" or "yeah" without permanently derailing the agent's thought).

4. Explore the Open-Source Lab

All reference code, architecture diagrams, and runnable barge-in experiments are open-sourced:

來源:DEV Community · dev.to