Openspline 試用：用人像照片和音訊生成即時説話頭像影片

I've built a few voice agents, and they always had the same problem: you talk to a blank screen.

So I tried Openspline, an open-source tool (MIT license) that takes a portrait photo plus audio and gives you back synchronized talking-head video, in real time. This is the walkthrough I wish I'd had, including the parts that tripped me up.

What it actually does

You give it two things: a portrait image and an audio stream. It gives you live video of that face speaking the audio. It doesn't care where the audio comes from. Your LLM, your TTS, and your prompts stay yours.

It has ready-made connectors for LiveKit, OpenAI Realtime, Gemini Live, ElevenLabs Agents, and Pipecat, plus a custom option for anything else that can produce audio.

Pick your path first

Worth knowing before you start: self-hosting needs Linux or WSL2 with an NVIDIA GPU. The README says AMD and Apple GPUs aren't supported yet, so Mac users should use the cloud.

Self-hosting in three steps

1. Install the GPU service

curl -fsSL https://raw.githubusercontent.com/openspline/openspline/main/install.sh | sh

It sets up a private Python environment and downloads the Low and High quality models.

2. Launch the demo

sh ~/.local/share/openspline/run.sh

Then open http://localhost:7860.
3. Install the SDK where your agent runs

pip install openspline

Python 3.10 to 3.12 is supported, and there's a Node SDK too (Node 20+).

Your first script

This is the core idea from the README:

from openspline import Openspline

client = Openspline()

async with client.avatar("portrait.jpg", quality="low") as avatar:
    print(avatar.viewer_url)
    await avatar.stream(agent.audio_stream(), sample_rate=24000)

avatar.viewer_url is a built-in page where the face appears. stream() takes generated PCM audio and returns when playback finishes. The README says WAV and MP3 files work too, and there's a file example in the repo (examples/python/universal.py), which is the easiest way to test without a full agent.

Low vs high quality

You can choose quality="low" (the default) or "high". The README's own short A100 tests measured roughly 64 FPS for low and 8 FPS for high, but it's explicit that these are engine speeds, not end-to-end playback latency. Since smooth motion needs inference to keep up with playback, I'd start with low.

Plugging it into a real agent

If you don't have a connector for your stack, the custom route is flexible. You push audio in chunks and signal what's happening:

await avatar.send_audio(chunk, sample_rate=24000)
await avatar.end_turn()      # when speech ends
await avatar.interrupt()     # on barge-in

That interrupt() call matters. Without it the avatar keeps talking after the user cuts in, which looks awful. The audio should be PCM16 by default, with options for float32 and MP3.

For popular stacks, the connectors wrap your existing event loop, so your tool calls and transcripts still reach your own handler.

Gotchas I'd tell a friend about

• Don't play audio twice. Openspline handles playback, so turn off your old speaker output.
• One session per worker. Each GPU worker serves one session at a time, and a busy worker returns a CapacityError. For more users you configure more workers in workers.yaml.
• The GPU stays busy for as long as a session is connected.
• Remote access needs extra setup. Use HTTPS for remote microphone access, and WebRTC playback may need open UDP ports or a TURN server.
• Keep your API key on the backend. Only pass avatar.session to a browser.
• Check the model licenses. The code is MIT, but the README says model weights have separate terms.
• It's young. The README itself notes that long-duration, multi-GPU, and some other validation is still incomplete.

Links

• Website: openspline.dev
• GitHub: openspline/openspline
• Docs: docs.openspline.dev

If you try it, tell me what you build, or what broke.