如何建立互動式音訊故事生成器

Why an Interactive Audio Story Generator?

Imagine a web app where users type a short prompt—“A brave rabbit discovers a hidden city”—and instantly get a narrated adventure, complete with character voices, background music, and dynamic branching. With modern voice‑AI and text‑to‑speech (TTS) tech, that fantasy is now a realistic project you can build over a weekend.

In this article you’ll learn:

• How to structure a story‑generation pipeline (text → audio).
• How to use ElevenLabs for high‑quality, expressive TTS and voice cloning.
• A minimal Python backend that talks to the ElevenLabs API.
• A simple JavaScript front‑end that lets users choose characters and hear the story in real time.

Grab your favorite editor, fire up a virtual environment, and let’s get coding.

1. Architecture Overview

+-----------------+       +----------------+       +-------------------+
|  Front‑end UI   |  -->  |  Story Engine  |  -->  |  ElevenLabs TTS   |
| (React/Vue…)   |       | (OpenAI / GPT) |       |  (audio files)    |
+-----------------+       +----------------+       +-------------------+
        ^                                                |
        |                                                v
   User interaction <-------------------------- Playback component

1. User input – The UI collects a prompt, character selections, and optional branching choices.
2. Story Engine – A language model (e.g., OpenAI’s gpt-4o-mini) expands the prompt into a short script, tagging each line with a speaker ID.
3. TTS Service – The script is sent to ElevenLabs, which returns an MP3 for each line.
4. Playback – The front‑end stitches the audio chunks together, optionally adding background ambience.

The heavy lifting (NLP & TTS) stays on the server, keeping the client lightweight and responsive.

2. Getting Started with ElevenLabs

ElevenLabs offers a REST API that can generate realistic speech from plain text. Their voice cloning feature lets you upload a few seconds of a speaker’s voice and get a custom model—perfect for giving each story character a distinct tone.

Pro tip: Sign up through this affiliate link to get a free credit boost: https://try.elevenlabs.io/kr07zfuqn1bp

You’ll receive an API key (keep it secret) and a list of voice IDs. For quick experiments you can use the built‑in “Rachel” voice (voice_id = "EXAVITQu4vr4xnSDxMaL").

3. Python Backend – From Prompt to Audio URLs

Below is a minimal Flask app that:

1. Calls OpenAI’s chat completion to turn a prompt into a scripted dialogue.
2. Sends each line to ElevenLabs for synthesis.
3. Returns a JSON array of temporary URLs that the front‑end can stream.

# app.py
import os
import uuid
import requests
from flask import Flask, request, jsonify
import openai

app = Flask(__name__)

ELEVEN_API_KEY = os.getenv("ELEVEN_API_KEY")
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
ELEVEN_TTS_URL = "https://api.elevenlabs.io/v1/text-to-speech"

openai.api_key = OPENAI_API_KEY


def generate_script(prompt: str) -> list[dict]:
    """Ask GPT to produce a short script with speaker tags."""
    response = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a story writer. Output a JSON array where each element has 'speaker' and 'line' keys. Keep it under 8 lines."},
            {"role": "user", "content": prompt},
        ],
    )
    # The model returns a JSON string; we parse it.
    import json
    return json.loads(response.choices[0].message.content)


def synthesize_line(text: str, voice_id: str) -> str:
    """Call ElevenLabs TTS and return a temporary URL."""
    payload = {
        "text": text,
        "voice_settings": {"stability": 0.75, "similarity_boost": 0.85},
    }
    headers = {
        "xi-api-key": ELEVEN_API_KEY,
        "Content-Type": "application/json",
    }
    r = requests.post(
        f"{ELEVEN_TTS_URL}/{voice_id}",
        json=payload,
        headers=headers,
        stream=True,
    )
    r.raise_for_status()

    # Save to a temporary file and serve it later (or use a cloud bucket)
    filename = f"temp/{uuid.uuid4()}.mp3"
    with open(filename, "wb") as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)

    # For demo purposes, we expose the file via Flask static route
    return f"/static/{os.path.basename(filename)}"


@app.route("/api/story", methods=["POST"])
def create_story():
    data = request.json
    prompt = data.get("prompt", "")
    script = generate_script(prompt)

    # Map speaker names to voice IDs (you could clone custom voices here)
    voice_map = {
        "Narrator": "EXAVITQu4vr4xnSDxMaL",   # default voice
        "Rabbit": "YOUR_CLONED_RABBIT_VOICE_ID",
        "Guardian": "YOUR_CLONED_GUARDIAN_VOICE_ID",
    }

    audio_urls = []
    for line in script:
        speaker = line["speaker"]
        voice_id = voice_map.get(speaker, "EXAVITQu4vr4xnSDxMaL")
        url = synthesize_line(line["line"], voice_id)
        audio_urls.append({"speaker": speaker, "url": url})

    return jsonify(audio_urls)


if __name__ == "__main__":
    os.makedirs("static", exist_ok=True)
    app.run(debug=True)

What’s happening?

• generate_script asks the language model to give you a structured script.
• synthesize_line posts the text to ElevenLabs (see the next section for a raw curl example) and writes the MP3 to a static folder.
• The endpoint returns an ordered list of URLs, ready for playback.

4. Calling ElevenLabs Directly with curl

If you just want to test the TTS endpoint, here’s a one‑liner:

curl -X POST "https://api.elevenlabs.io/v1/text-to-speech/EXAVITQu4vr4xnSDxMaL" \
  -H "xi-api-key: YOUR_ELEVENLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "text": "Hello, brave explorer! Your adventure begins now.",
        "voice_settings": {"stability":0.7,"similarity_boost":0.9}
      }' --output hello.mp3

Replace EXAVITQu4vr4xnSDxMaL with any voice ID you’ve created, and you’ll have an MP3 file (hello.mp3) on your disk.

5. Front‑End: Streaming the Story

A tiny React component can fetch the JSON from /api/story and play each clip sequentially using the Web Audio API.

// StoryPlayer.jsx
import { useState } from "react";

export default function StoryPlayer() {
  const [prompt, setPrompt] = useState("");
  const [clips, setClips] = useState([]);
  const [playing, setPlaying] = useState(false);

  const start = async () => {
    const res = await fetch("/api/story", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ prompt }),
    });
    const data = await res.json();
    setClips(data);
    playSequence(data);
  };

  const playSequence = async (list) => {
    setPlaying(true);
    for (const { url } of list) {
      const audio = new Audio(url);
      await new Promise((resolve) => {
        audio.onended = resolve;
        audio.play();
      });
    }
    setPlaying(false);
  };

  return (
    <div>
      <textarea
        rows={3}
        placeholder="Enter your story prompt..."
        value={prompt}
        onChange={(e) => setPrompt(e.target.value)}
      />
      <button onClick={start} disabled={playing || !prompt}>
        {playing ? "Playing…" : "Generate & Play"}
      </button>
    </div>
  );
}

The component:

1. Sends the user prompt to the Flask backend.
2. Receives an ordered list of audio URLs.
3. Plays each clip one after another, giving the illusion of a single, seamless narration.

You can extend this by adding background ambience (e.g., rain, forest sounds) using the gainNode of the Web Audio API, or by letting users pick alternate voices from a dropdown that maps to different cloned IDs.

6. Voice Cloning – Giving Characters Their Own Identity

ElevenLabs lets you upload 30‑second voice samples and instantly receive a voice_id. Here’s a quick workflow:

1. Record a short script for each character (you can use Audacity or even a phone).
2. Use the Voice Cloning endpoint:

curl -X POST "https://api.elevenlabs.io/v1/voices/add" \
  -H "xi-api-key: YOUR_ELEVENLABS_API_KEY" \
  -F "name=Rabbit" \
  -F "files=@rabbit_sample.wav"

The response contains a new voice_id. Store this ID in your voice_map dictionary from the Python example, and every line spoken by “Rabbit” will now sound like your custom voice.

7. Scaling Up – Tips for Production

8. Wrap‑Up

You now have a full‑stack blueprint for an interactive audio story generator:

• Prompt → script with a language model.
• Script → voice using ElevenLabs’ TTS and cloning capabilities.
• Audio → playback in the browser with a lightweight JavaScript player.

The core pieces are only a few dozen lines of code, yet the result feels magical—especially when you give each character a unique, human‑like voice.

Ready to turn your own ideas into spoken adventures? Grab an ElevenLabs account through the affiliate link below, spin up the Flask server, and start building. The only limit is your imagination.

Try ElevenLabs now and bring your stories to life: https://try.elevenlabs.io/kr07zfuqn1bp