我如何為 Fud AI Live Voice 建立一套工具契約，接入六家即時語音服務商

Fud AI 8.0 is out on iPhone and iPad (Android is in Google Play review) with Live Voice: you talk to the app, and it logs meals, scans barcodes, checks your macros and edits your diary while you watch it happen on screen.

Here's how it works under the hood. Fud AI is open source (MIT), so every file below is public.

One tool contract, three codebases

The assistant can only act through tools. They live in one file, shared/voice-assistant/tools.json: 25 actions like open_camera, capture_photo, scan_barcode, log_food, log_water, start_fast, log_workout, update_setting and end_conversation.

The iOS app, the Android app and the Cloudflare Worker all follow that contract. Both apps bundle an identical copy, and a CI script fails the build if either copy drifts. I change the JSON first, then each platform.

Every tool also declares its effect: read, navigate, write or destructive. The UI behaviour comes from that:

• Writes show a card with Undo for 8 seconds.
• Destructive actions always need confirmation.
• With "Confirm before saving" on (the default), a write without user_confirmed: true changes nothing. It returns confirmation_required, the app shows a Confirm/Cancel card, and the model has to ask you before calling again.

The model is also told never to invent numbers. Calories and macros come from the data tools, not from its memory.

Six realtime providers, one experience

Live Voice runs on whichever provider you choose:

They differ more than you'd expect:

• Audio: Gemini, Qwen and hosted take 16 kHz PCM in and send 24 kHz out. OpenAI and xAI use 24 kHz both ways.
• Tool calls: Gemini sends toolCall.functionCalls and wants toolResponse. OpenAI, xAI and Qwen send response.function_call_arguments.done and want a function_call_output item followed by response.create.
• Vision: Gemini, Qwen and hosted get one camera frame per second while the camera is open. OpenAI gets the latest frame with each turn. xAI and GPT-Live get none, so the model asks you to say when the food is in frame.

Each provider gets a thin adapter, and everything above it (tools, cards, confirmation, undo) is shared. Own keys go in headers (x-goog-api-key or Authorization: Bearer), never in URLs.

The hosted relay

For people who don't want to manage an API key, the Plus/Pro plan connects to a WebSocket relay on Cloudflare Workers that forwards to Gemini Live. A relay like that could easily turn into a free general-purpose Gemini key, so it's strict:

• Before the upgrade, the Worker checks the user id, rate limits, the RevenueCat plan and the remaining voice balance.
• The first message must be setup. The Worker rebuilds it from an allow-list: the model is pinned server-side, tools are function declarations only, and instructions are size-limited.
• After that, only realtimeInput, clientContent and toolResponse are accepted, each 1 MiB at most, with a bounded input rate.
• Billing is by time: 15-second blocks, each charged when it starts. Sessions end at 15 minutes.
• Nothing you say or show is stored. Logs carry structured codes and a hashed id prefix only.

The orb

The glowing orb is a Metal shader on iOS and an AGSL port on Android 13+, with a Canvas fallback on older versions. It's driven by the loudness of whoever is talking, so you can see when it's listening, thinking or speaking.

You can start a session from the orb on Home, Siri, the Action Button or Control Center on iPhone, or a Quick Settings tile, shortcut or widget on Android.

Try it

Fud AI is free on iOS and Android, and Live Voice reaches Android as soon as Google approves 8.0. Use your own AI key, or the optional Plus/Pro plan if you'd rather not. No account needed.

• Website: https://fud-ai.app
• Source: https://github.com/apoorvdarshan/fud-ai

I'd love feedback on the voice experience, and PRs are welcome.