Timeline Studio 以 Skills 和 MCP 支援 AI 智能體編輯影片時間軸

An AI agent can generate an FFmpeg command. But what happens when the next request is: “Shorten the second shot, move the voiceover and its captions together, and give me a project I can keep editing”?

That is the problem I have been working on in Timeline Studio, an open-source, local-first browser video editor with an Agent Skill and a shared command layer.

• Source code
• Try the editor

This post focuses on the editing architecture: how an agent turns a request into an inspectable plan, how that plan changes project state, and where rendering still has explicit limits.

Start with an editable project

A rendered MP4 does not preserve the decisions needed for the next edit: clip identity, source ranges, timing, transforms, and caption/audio relationships.

Timeline Studio uses an editable .timeline project as the source of truth. In the local command workflow, the archive contains project.json and embedded media. Asset import probes necessary metadata, computes SHA-256 integrity information, and embeds the media bytes.

This gives both the user and the agent a concrete state to return to. The user can reopen the project in the editor. The agent can inspect it before making another change.

Separate editorial decisions from state changes

The model can decide that a voiceover should start at three seconds. The application should enforce whether the clip exists, whether the time is valid, and how linked captions follow it.

The architecture looks like this:

User request + media evidence
             |
     Skill: analyze and plan
             |
     Structured edit plan
             |
 CLI / local MCP / browser WebMCP
             |
 Shared command registry and reducers
             |
 Editable project + verified output

The transports have different environments. The CLI operates on local archives. The local STDIO MCP adapter exposes that command runner to an agent. Browser WebMCP, when exposed by the host, operates on the open editor and commits through its normal history.

The important constraint is to reuse editing rules rather than implement a separate editor for each transport.

An edit plan needs preconditions

Here is a simplified plan that moves an existing voiceover to project time 3s:

{
  "schemaVersion": 1,
  "project": "/projects/input.timeline",
  "baseRevision": 12,
  "dryRun": false,
  "operations": [
    {
      "id": "move-voice-001",
      "type": "timed.move",
      "track": "audio",
      "clipId": "voice-1",
      "start": 3
    }
  ],
  "output": {
    "project": "/projects/output.timeline"
  }
}

The revision and clip ID are illustrative: they must come from inspecting the actual project.

Each field has a job:

• baseRevision checks the state on which the plan was based.
• Operation IDs support idempotent handling of retries.
• clipId identifies an existing project object.
• type restricts the action to a supported operation.
• output.project declares a new output artifact.

For this audio move, still-linked captions move by the same delta. That behavior belongs in the editing rules.

Preview a semantic diff before applying

The workflow is:

inspect -> plan -> validate -> diff -> apply -> inspect

The CLI exposes it directly:

npm run agent -- project.inspect /projects/input.timeline

node skills/edit-timeline-studio/scripts/validate_edit_plan.mjs \
  /projects/plan.json

npm run agent -- project.diff /projects/plan.json
npm run agent -- project.run /projects/plan.json

The standalone validator checks transport shape, finite time fields, and duplicate operation IDs. It does not prove that a plan is valid for a particular project.

project.diff is the project-aware dry run. It uses the same command registry and reducers as application, without writing an archive. It checks supported operations and preconditions, then reports project fields, clip additions/removals, changed fields, ordering, and duration changes.

This makes accidental scope expansion visible. A request to move narration should not quietly change the visual sequence duration.

Retries and concurrent edits are normal cases

Three mechanisms matter here.

Idempotency

If an operation completes but its response is lost, retrying the same operation ID should not insert the same clip twice. Previously applied IDs are treated as no-ops.

Revision checks

If the agent inspected revision 12 and the user subsequently edited the project, a stale plan containing new operations must fail with a revision conflict. The agent needs to inspect again and reconsider the plan.

Transactional application

A failed batch should not leave a partially edited output archive. Local CLI and MCP writes also require a new output path and reject overwriting the input or an existing output.

These are familiar application-engineering techniques. They become especially useful when the caller is an agent that may retry or reason from outdated state.

Video editing has more than one clock

A timestamp can refer to source media, a clip-local position, or the final project.

For a constant-speed clip:

sourceTime = sourceIn + (timelineTime - clipStart) * speed

With a speed curve, source-time mapping must accumulate the changing playback speed over time.

An agent identifying an action at source time 8s cannot assume that the cut belongs at project time 8s. Trimming, splitting, keyframes, thumbnails, and effect analysis all depend on maintaining that mapping.

Tracks also have different semantics. Main visuals form a contiguous sequence; narration, captions, and overlays have explicit timing. Whether a duration change shifts other tracks depends on ripple mode, links, and track locks.

These rules need to be explicit application behavior.

Skills and MCP serve different purposes

The Skill describes the workflow: inspect material, plan the edit, choose an execution path, and verify the result.

The MCP adapter exposes concrete tools, including:

timeline_project_inspect
timeline_track_inspect
timeline_clip_inspect
timeline_project_diff
timeline_project_apply
timeline_project_render

The local adapter is a thin transport over the existing command runner. It does not duplicate timeline editing logic.

To install the Skill:

npx skills add MartinDelophy/ai-video-editor \
  --skill edit-timeline-studio

To explore the repository and local MCP server:

git clone https://github.com/MartinDelophy/ai-video-editor.git
cd ai-video-editor
npm install
npm run mcp

Installing the Skill does not install the complete runtime. Local MCP needs the checkout and Node dependencies; media operations need their corresponding tools. The Skill includes a host dependency-check workflow.

An edited project is not yet a verified video

The current local project.render path supports the documented portable Visuals, Voiceover, and Music subset. It uses FFmpeg to produce H.264/AAC MP4 and ffprobe to check dimensions, duration, and audio-track presence.

It rejects unsupported composition features rather than silently dropping visible captions, overlays, transitions, or effects. Richer compositions require the browser renderer.

Verification needs several levels: inspect the project, reopen it and resolve archived media, check the output container, and actually play the important visual and audio boundaries. A valid project structure alone cannot prove that the final mix sounds right.

Where to read the implementation

Useful entry points in the repository:

• src/lib/projectCommandEngine.js — shared project command engine
• scripts/timeline-command.mjs — local archive operations and rendering entry point
• skills/edit-timeline-studio/mcp/server.mjs — STDIO MCP adapter
• src/hooks/useWebMcpEditor.js — browser editor integration
• skills/edit-timeline-studio/SKILL.md — agent workflow

The command reference documents the current contract and limits.

The larger lesson for me is that agent reliability depends on the application interface: explicit state, constrained operations, reviewable changes, and verifiable outputs.

If you are building agent tools or working on browser media, I would welcome feedback on the architecture. Real editing failures and reproducible cases are particularly useful. You can explore the source, try the editor, or browse the editing handbook.

Writing disclosure: this article was drafted with AI assistance using the project's source code and documentation.