跳到正文
DEV Community· heavy shield·· 2 小時前AI 評分36

Claude 在 Blender 指揮製作完整 YouTube 影片所需的技術架構

The Stack I'd Need for Claude to Direct a Whole YouTube Video in Blender

AI 導讀

作者搭建一套 Python 技術架構，讓 Claude 透過工具型智能體連接自然語言與 Blender Python API，目標是在無需人工介入下製作完整 YouTube 影片。架構包含在 Blender 內運行的 JSON-RPC 伺服器、工具定義、智能體循環及渲染工具，並以白名單操作取代執行任意程式碼。

正文

Introduction

I've been experimenting with LLM agents that control Blender. The goal: have Claude generate a complete YouTube video—script, 3D scenes, animations, camera moves, and render—without human intervention. The result is a stack of Python components that bridge natural language to Blender's Python API. This post outlines the architecture, the code, and the hard limits I hit.

Problem

Blender is a GUI application with a massive Python API (bpy). An LLM can emit Python code, but directly executing it is dangerous and brittle. You need:

  • Safety: Arbitrary code execution can delete files or crash the system.
  • State management: Blender scenes are complex; the agent must track objects, materials, and timelines.
  • Iterative refinement: A single prompt won't produce a polished video. The agent must plan, execute, observe, and fix.
  • Rendering: Final output requires invoking Blender's render engine, which can take hours.

Naively asking Claude to "write a Blender script" yields code that often fails due to API changes or missing context.

Solution

I built a tool-using agent loop (inspired by ReAct-style prompting) where Claude calls a set of Python functions that manipulate Blender via bpy. The agent runs inside Blender's Python environment (or a subprocess with blender --background). The loop terminates when the agent calls a finish tool or exceeds a step limit.

Key components:

  1. Blender Bridge: A JSON-RPC server running inside Blender that executes whitelisted operations.
  2. Tool Definitions: Python functions decorated with metadata for Claude's tool-use API.
  3. Agent Loop: Sends conversation + tool results to Claude, parses tool calls, executes them, and appends results.
  4. Renderer: A final tool that triggers bpy.ops.render.render(animation=True).

This avoids eval of arbitrary code. Each tool is a specific, auditable function.

Implementation

1. Blender Bridge (inside Blender)

Run this script in Blender's scripting tab or via blender --background --python bridge.py. It listens on a local socket for JSON commands.

# bridge.py
import bpy
import json
import socket
import threading

HOST = '127.0.0.1'
PORT = 65432

# Whitelisted operations
def op_create_cube(name, location):
    bpy.ops.mesh.primitive_cube_add(location=location)
    obj = bpy.context.active_object
    obj.name = name
    return {'name': obj.name, 'location': list(obj.location)}

def op_set_camera(location, rotation):
    cam = bpy.context.scene.camera
    if not cam:
        bpy.ops.object.camera_add()
        cam = bpy.context.active_object
        bpy.context.scene.camera = cam
    cam.location = location
    cam.rotation_euler = rotation
    return {'camera': cam.name}

def op_render(output_path, frame_start=1, frame_end=1):
    bpy.context.scene.frame_start = frame_start
    bpy.context.scene.frame_end = frame_end
    bpy.context.scene.render.filepath = output_path
    bpy.ops.render.render(animation=True)
    return {'rendered': output_path}

OPERATIONS = {
    'create_cube': op_create_cube,
    'set_camera': op_set_camera,
    'render': op_render,
}

def handle_client(conn):
    with conn:
        data = conn.recv(4096).decode('utf-8')
        if not data:
            return
        try:
            msg = json.loads(data)
            op = msg.get('operation')
            args = msg.get('args', {})
            if op not in OPERATIONS:
                response = {'error': f'Unknown operation: {op}'}
            else:
                result = OPERATIONS[op](**args)
                response = {'result': result}
        except Exception as e:
            response = {'error': str(e)}
        conn.sendall(json.dumps(response).encode('utf-8'))

def start_server():
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.bind((HOST, PORT))
        s.listen()
        print(f'Bridge listening on {HOST}:{PORT}')
        while True:
            conn, _ = s.accept()
            threading.Thread(target=handle_client, args=(conn,)).start()

# Start server in a background thread so Blender's UI remains responsive
threading.Thread(target=start_server, daemon=True).start()

Security note: This bridge executes only whitelisted functions. Never expose it to untrusted networks. Bind to 127.0.0.1 and use authentication if needed.

2. Tool Definitions for Claude

Define tools in a format compatible with Anthropic's tool-use API.

# tools.py
tools = [
    {
        "name": "create_cube",
        "description": "Create a cube mesh at a given location",
        "input_schema": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "location": {"type": "array", "items": {"type": "number"}, "minItems": 3, "maxItems": 3}
            },
            "required": ["name", "location"]
        }
    },
    {
        "name": "set_camera",
        "description": "Set the active camera's location and rotation (Euler angles in radians)",
        "input_schema": {
            "type": "object",
            "properties": {
                "location": {"type": "array", "items": {"type": "number"}, "minItems": 3, "maxItems": 3},
                "rotation": {"type": "array", "items": {"type": "number"}, "minItems": 3, "maxItems": 3}
            },
            "required": ["location", "rotation"]
        }
    },
    {
        "name": "render",
        "description": "Render an animation to the given output path",
        "input_schema": {
            "type": "object",
            "properties": {
                "output_path": {"type": "string"},
                "frame_start": {"type": "integer"},
                "frame_end": {"type": "integer"}
            },
            "required": ["output_path"]
        }
    }
]

3. Agent Loop

This loop sends messages to Claude, executes tool calls via the bridge, and appends results. It terminates when Claude calls a finish tool (not shown) or after max_steps.

# agent.py
import json
import socket
import anthropic

client = anthropic.Anthropic(api_key="YOUR_API_KEY")

def call_bridge(operation, args):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect(('127.0.0.1', 65432))
        s.sendall(json.dumps({'operation': operation, 'args': args}).encode('utf-8'))
        response = s.recv(4096).decode('utf-8')
        return json.loads(response)

def run_agent(prompt, max_steps=20):
    messages = [{"role": "user", "content": prompt}]
    for step in range(max_steps):
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=1024,
            tools=tools,
            messages=messages
        )
        # Append assistant's response
        messages.append({"role": "assistant", "content": response.content})
        tool_calls = [block for block in response.content if block.type == "tool_use"]
        if not tool_calls:
            print("No tool calls, ending.")
            break
        for tool_call in tool_calls:
            if tool_call.name == "finish":
                print("Agent finished.")
                return
            result = call_bridge(tool_call.name, tool_call.input)
            messages.append({
                "role": "user",
                "content": [{
                    "type": "tool_result",
                    "tool_use_id": tool_call.id,
                    "content": json.dumps(result)
                }]
            })
    else:
        print(f"Reached max steps ({max_steps}).")

4. Orchestration

Start Blender with the bridge, then run the agent from a separate process.

blender --background --python bridge.py &
python agent.py

Key Takeaways

  • Tool whitelisting beats code generation: Instead of letting Claude write arbitrary bpy code, expose a small set of safe operations. This prevents catastrophic mistakes and makes debugging easier.
  • Explicit termination: The agent loop stops on a finish tool call or a step limit. Without this, it could run indefinitely.
  • Security first: Never eval LLM output. My bridge uses a whitelist and runs locally. If you must execute generated code, sandbox it (e.g., RestrictedPython or a container).
  • Iterative refinement is key: The agent can observe results (e.g., object locations) and adjust. For a full video, you'd add tools for keyframing, text, and compositing.
  • Rendering is the bottleneck: Blender renders can take hours. The agent should plan the entire scene before triggering a render.

This stack is a starting point. With more tools (e.g., add_keyframe, set_material), Claude can direct increasingly complex animations. The code is available on GitHub—link in comments.

來源：DEV Community · dev.to

#教學/實踐#Agent#MCP/工具調用#Anthropic