跳到正文
DEV Community· VoiceDeveloper·· 1 小時前AI 評分39

如何為影片內容建立 AI 配音工具

Create an AI Dubbing Tool for Video Content

AI 導讀

這篇教學示範如何用 Python 和 ElevenLabs 建立影片 AI 配音流程,將新語音軌合成並配回影片。流程包括擷取原音、選擇性轉錄及翻譯、生成新語音、調整音訊長度,以及將配音與影片合併。

正文

Why an AI Dubbing Tool?

If you’ve ever tried to localize a video, you know the pain: hiring voice actors, coordinating recording sessions, and then painstakingly syncing the new audio back to the footage. With modern text‑to‑speech (TTS) and voice‑cloning services, you can automate most of that workflow. In this article we’ll build a lightweight AI dubbing pipeline that takes a source video, generates a new voice track in any language you like, and stitches it back together—all with a few lines of Python and a call to ElevenLabs.

TL;DR – Extract audio → transcribe (optional) → generate new speech → align & mux → 🎉


What You’ll Need

Item Reason
Python 3.9+ Main scripting language
ffmpeg Extract and re‑encode audio/video
pydub Simple audio manipulation
requests Call ElevenLabs REST API
(Optional) whisper or any speech‑to‑text model If you want to translate the script before dubbing
An ElevenLabs API key The magic behind high‑quality voice synthesis

Install the Python dependencies:

pip install pydub requests tqdm
# optional for transcription
pip install openai-whisper

Make sure ffmpeg is in your PATH – most OS package managers have it (brew install ffmpeg, apt-get install ffmpeg, etc.).


Step 1 – Pull the Original Audio

First we need the raw audio track from the video. ffmpeg can do this in a single command:

ffmpeg -i input_video.mp4 -vn -acodec pcm_s16le -ar 44100 -ac 2 original.wav
  • -vn disables video output.
  • pcm_s16le gives us a lossless WAV file that works well with most TTS services.

If you prefer to keep everything inside Python, subprocess works just as well:

import subprocess

def extract_audio(video_path, wav_path):
    cmd = [
        "ffmpeg",
        "-i", video_path,
        "-vn",
        "-acodec", "pcm_s16le",
        "-ar", "44100",
        "-ac", "2",
        wav_path,
        "-y"  # overwrite if exists
    ]
    subprocess.run(cmd, check=True)

extract_audio("input_video.mp4", "original.wav")

Step 2 – (Optional) Transcribe & Translate

If you already have a script, skip this step. Otherwise, you can run an automatic transcription and then translate it with any LLM or translation API. Here’s a quick Whisper example:

import whisper

model = whisper.load_model("base")
result = model.transcribe("original.wav")
script = result["text"]
print("Transcribed script:", script)

You could pipe script into Google Translate, DeepL, or even an OpenAI LLM to get the target language text.


Step 3 – Generate New Speech with ElevenLabs

ElevenLabs offers a REST endpoint that can produce lifelike speech in a matter of seconds. The following function sends the text and streams back a WAV file.

import requests
from pathlib import Path

ELEVEN_API_KEY = "YOUR_ELEVENLABS_API_KEY"
VOICE_ID = "EXAVITQu4vr4xnSDxMaL"  # default “Rachel” voice – replace with your cloned voice ID

def synthesize(text: str, out_path: Path):
    url = f"https://api.elevenlabs.io/v1/text-to-speech/{VOICE_ID}"
    headers = {
        "xi-api-key": ELEVEN_API_KEY,
        "Content-Type": "application/json",
        "Accept": "audio/wav"
    }
    payload = {
        "text": text,
        "model_id": "eleven_monolingual_v1",
        "voice_settings": {
            "stability": 0.75,
            "similarity_boost": 0.85
        }
    }

    response = requests.post(url, json=payload, headers=headers, stream=True)
    response.raise_for_status()
    with open(out_path, "wb") as f:
        for chunk in response.iter_content(chunk_size=8192):
            f.write(chunk)

synthesize(script, Path("dub.wav"))

Pro tip: If you’ve trained a custom voice on ElevenLabs (they call it voice cloning), just replace VOICE_ID with the ID you received after the cloning process. The same API call works for both stock and cloned voices.


Step 4 – Align the New Audio to the Video

The naive approach is to assume the new script has the same duration as the original. In practice, you’ll need to time‑stretch or pad the audio to match the video’s timeline. pydub makes this straightforward:

from pydub import AudioSegment
from pydub.utils import mediainfo

def match_duration(source_wav, target_wav):
    src = AudioSegment.from_wav(source_wav)
    tgt = AudioSegment.from_wav(target_wav)

    src_len = len(src)   # milliseconds
    tgt_len = len(tgt)

    if tgt_len > src_len:
        # Trim excess
        tgt = tgt[:src_len]
    else:
        # Pad with silence at the end
        silence = AudioSegment.silent(duration=src_len - tgt_len)
        tgt = tgt + silence

    tgt.export("dub_aligned.wav", format="wav")
    return "dub_aligned.wav"

aligned_path = match_duration("original.wav", "dub.wav")

If you need finer control (e.g., per‑sentence alignment), you can use forced‑alignment tools like Aeneas or Montreal Forced Aligner, but that’s beyond the scope of this quick demo.


Step 5 – Mux the Dubbed Audio Back Into the Video

Now that the audio length matches the video, we simply replace the original track:

ffmpeg -i input_video.mp4 -i dub_aligned.wav -c:v copy -map 0:v:0 -map 1:a:0 -shortest output_dubbed.mp4
  • -c:v copy tells ffmpeg to keep the video stream unchanged.
  • -shortest ensures the output stops when the shortest stream ends (useful if you have tiny mismatches).

You can wrap this in Python as well:

def mux(video_path, audio_path, out_path):
    cmd = [
        "ffmpeg",
        "-i", video_path,
        "-i", audio_path,
        "-c:v", "copy",
        "-map", "0:v:0",
        "-map", "1:a:0",
        "-shortest",
        out_path,
        "-y"
    ]
    subprocess.run(cmd, check=True)

mux("input_video.mp4", aligned_path, "output_dubbed.mp4")

That’s it—your video now speaks in a brand‑new voice!


Bonus: Automating the Whole Pipeline

Putting everything together in a single script makes it reusable:

def dub_video(
    video_file: str,
    target_text: str,
    output_file: str,
    voice_id: str = VOICE_ID,
):
    extract_audio(video_file, "original.wav")
    synthesize(target_text, Path("dub.wav"))
    aligned = match_duration("original.wav", "dub.wav")
    mux(video_file, aligned, output_file)

# Example usage
dub_video(
    video_file="input_video.mp4",
    target_text=script,          # could be the original script or a translated version
    output_file="final_dub.mp4"
)

Feel free to expand it with command‑line arguments, batch processing, or a tiny Flask API if you want a web‑based dubbing service.


Tips for Production‑Ready Dubbing

  1. Voice Consistency – Clone a single voice for an entire series so viewers recognize the narrator.
  2. Pronunciation Tweaks – ElevenLabs accepts SSML (Speech Synthesis Markup Language). Wrap tricky words in <phoneme alphabet="ipa" ph="..."> to force correct pronunciation.
  3. Background Noise – If the original video has ambient sounds, consider mixing a low‑level version of the original audio under the new voice to preserve atmosphere.
  4. Legal – Always respect copyright and obtain proper rights when dubbing third‑party content.

Wrap‑Up

You’ve just built a functional AI dubbing tool using open‑source utilities and the powerful ElevenLabs TTS platform. From extracting audio to syncing the final track, the whole workflow can be scripted in under a minute per video, opening doors to rapid localization, prototype voice‑overs, or even personalized video messages.

Ready to give your videos a fresh voice? Grab your ElevenLabs API key, follow the steps above, and start dubbing today! 🚀

Try ElevenLabs now and see how easy high‑quality voice cloning can be: https://try.elevenlabs.io/kr07zfuqn1bp.

來源:DEV Community · dev.to