我的 Google Voice AI 智能體如何判斷何時開口：通話狀態偵測、Groq 與 TTS

Most AI voice demos start with a telephony API. You get webhooks, clean call events and an audio stream, and the interesting problem is the conversation.

My Google Voice Dispatch Agent doesn't get any of that. It drives Google Voice inside a real Chrome window, and that changes where the hard work is. The AI reply turned out to be the easy part. The hard part was knowing, with confidence, whether someone had actually picked up.

This post walks through how the agent is put together, based on what's in the repo.

Repo: https://github.com/mafzalkalwardev/google-voice-dispatch-agent (MIT licensed)

What it does

In short, it's an outbound AI calling console for Windows:

Selenium controls Google Voice using a persistent Chrome profile and dials contacts from a CSV/XLSX list or CRM data.
It watches the call state: ringing, connected, ended, failed, voicemail.
Once a call is really connected, a realtime loop runs: voice activity detection, speech-to-text, an LLM reply via Groq, and text-to-speech played back into the call.
Voicemail is detected from DOM cues plus an early-call audio classifier, and the agent hands off to a pre-recorded voicemail message.
Every call is logged with transcript, recording, outcome and CRM update.
A FastAPI web console covers settings, preflight checks, audio tools, live runs, logs, recordings and leads. It's also packaged as a Windows EXE with PyInstaller.

The runtime flow

The README describes the flow in ten steps, and I'll keep it close to that:

Load .env, dialer_config.json, contacts and CRM data.
Run preflight checks for Groq, the Chrome profile, contacts, callback number and audio devices.
Launch Google Voice with the configured Chrome profile.
Prepare the opening line and voicemail audio before dialing.
Dial one contact.
Poll detect_call_state() until connected, voicemail, no answer, ended or failed.
Start the realtime conversation loop only after confirmed connection.
Watch for voicemail during the early connected window and hand off if confirmed.
Save logs, transcript, recording, CRM update and call archive.
Cool down, then move to the next contact.

Step 4 is small but matters a lot. If you generate the greeting after someone says "hello", they hear dead air and hang up. Pre-generating it means the agent can talk the moment the call is confirmed.

Call-state detection: the real product

Here's the thing about automating a web UI: the DOM is noisy. Buttons appear early, old "call ended" banners linger, and voicemail-looking cues can show up while the phone is still ringing. If you trust the first signal you see, the bot either talks over the ring tone or hangs up on a real person.

So the core of the project is a conservative gate in src/google_voice.py. The rules it enforces:

Ringing can't instantly jump to ended.
min_ring_seconds has to pass before connected, ended or voicemail transitions are accepted.
Active call controls that appear too early are logged, and the state is held at ringing.
Stale ended banners are ignored during the minimum ringing window.
Voicemail cues are ignored while still ringing.
Connected needs stable evidence, like answered controls or a call timer.
The AI opening line starts only after confirmed connection.
max_ring_seconds ends the attempt as no answer only after the ring timeout.

Every decision is logged too: current state, elapsed ring time, DOM cues, answered controls, call-active status, voicemail cues, the audio classifier result and the timeout reason. When a call goes wrong, I can read exactly why the agent thought what it thought.

The design principles in the README sum it up:

Never speak until the call is actually connected.
Never treat immediate DOM noise as a completed call.
Respect a minimum ringing duration.
Detect voicemail only after connected/greeting evidence.

The conversation loop: VAD, Groq and TTS

Once the gate says "connected", conversation_loop.py takes over. The modules split up cleanly:

src/
  google_voice.py          Selenium Google Voice automation and call-state detection
  call_session.py          Call session model and CallState transitions
  conversation_loop.py     Realtime capture, VAD, STT, LLM, TTS, voicemail handoff
  voicemail_detector.py    Audio classifier for greeting/beep/live-call signals
  ai_groq.py               Groq-backed AI response behavior
  realtime_tts.py          Edge TTS playback and cache integration
  audio_capture.py         Input capture and VAD framing
  audio_routing.py         Windows audio-device discovery and playback routing

The loop works like this:

Capture and VAD. audio_capture.py captures the caller's audio and frames it for voice activity detection, so the agent knows when the person has started and stopped talking.
Speech-to-text and reply. The speech goes to Groq STT, and ai_groq.py generates the reply.
Text-to-speech. realtime_tts.py speaks it with Edge TTS, with caching, and pyttsx3 as a fallback.
Routing. The voice has to get into the browser call, which is where VB-CABLE comes in.

Several knobs in dialer_config.json relate directly to this loop:

{
  "min_ring_seconds": 2,
  "max_ring_seconds": 45,
  "voicemail_detect_seconds": 15,
  "silence_does_not_end_call": true,
  "call_cooldown_seconds": 10,
  "tts_warmup": true,
  "stt_retry_count": 2,
  "vad_silence_frames": 12,
  "vad_speech_frames": 2
}

silence_does_not_end_call is a nice example of a lesson baked into config: people go quiet on the phone, and that shouldn't be read as a hang-up.

Audio routing is half the battle

The minimum VB-CABLE setup from the README:

Agent playback device: CABLE Input
Chrome microphone input: CABLE Output
Chrome speaker output: a device the agent can capture

For unattended runs, the README recommends separate routes for TTS and inbound capture, and warns against capturing the same speaker output that plays the agent's voice. Otherwise the agent can end up transcribing itself. CAPTURE_DEVICE=default uses Windows speaker loopback, which is fine for testing but less reliable for nonstop calling.

There are CLI helpers for checking this before going live:

python -m src.main --list-audio-devices
python -m src.main --audio-route-test
python -m src.main --preflight

Parallel lines without crossed wires

The agent can also run more than one line. Each worker gets its own Chrome profile and pulls from one shared contact queue. The important safety rule: shared audio routes are rejected by default, "because one AI response would otherwise be broadcast into multiple calls." Each slot needs its own loopback and capture pair, like CABLE-A and CABLE-B.

Google Voice accounts can be managed from the console. Saved passwords are protected with Windows DPAPI and never returned by the web API, and MFA or CAPTCHA challenges still have to be completed by a human in the visible Chrome window.

What I'd tell someone building something similar

Treat state detection as the main feature. Spend your time there before tuning prompts.
Log your reasoning, not just outcomes. "Connected" is useless when debugging. "Connected because the timer appeared at 6.2s after ringing" is useful.
Prepare audio ahead of time. Silence at pickup kills calls.
Isolate audio per line. Crossed audio is far worse than a slower dialer.
Add preflight checks. Most failed runs I saw were config problems, not code bugs.

And please: anything that dials real people needs consent, Do Not Call compliance and call-recording disclosure where required. The README also recommends keeping min_ring_seconds above zero, using a cooldown, and not rapid-firing failed numbers.

If you're working on voice agents on top of tools you already use, I'd like to hear what broke first for you.

Open to freelance & remote work
Portfolio: https://mafzalkalwardev.github.io/#hire-me
GitHub: https://github.com/mafzalkalwardev
WhatsApp: +92 307 9670503 — https://wa.me/923079670503