Dynamo 為編碼智能體提供視覺觀測，並為 PR 審查生成佐證

What I Built

When an AI coding agent or browser-use sandbox fails in CI at 2:00 AM, everyone flies blind. The agent acts on a headless display without visual feedback, the test runner spits out a 400-line stack trace, and the human reviewer gets a single static PNG that doesn't show which action triggered the regression.

If you try to fix this by feeding raw screen recordings into a Vision-Language Model (VLM), you blow your token budget in under a minute. If you hack together a bash script around Xvfb and ffmpeg, a sudden exception sends SIGKILL, corrupts the MP4 moov atom, and leaves you with zero artifacts when you need them most.

Dynamo is a headless, zero-dependency Python visual and video harness for coding agents, Playwright browser automation, and computer-use sandboxes. In a single with VisualHarness(): context manager (or a single CLI command), Dynamo:

1. Manages the headless X11 lifecycle (Xvfb) with a socket readiness probe so your browser never races the display server.
2. Streams non-blocking screen capture (ffmpeg -f x11grab) with graceful SIGINT finalization — even when your agent crashes mid-run.
3. Logs timestamped action telemetry (telemetry.jsonl) and burns visual click-rings and crosshairs directly into MP4 and looping PR GIFs.
4. Answers visual questions with model-free arithmetic — cropping before encoding, pixel OCR on <canvas> elements, visual diffs, and Set-of-Marks element grounding.
5. Keeps only the frames that matter — scene changes anchored to exact action timestamps — instead of paying for a thousand-odd near-identical frames.
6. Routes surviving keyframes through local open-weight models (qwen3-vl:4b under Apache-2.0 and laya for lightweight typed filtering) so proprietary screens never leave your infrastructure.

Above: A live headless Chromium session recorded inside Dynamo's virtual display (:99), compiled into a two-pass palette GIF with telemetry click rings and action timestamps baked in.

🎬 Video Demo & Visual Showcase

Here is the full product walkthrough demonstrating Dynamo's architecture, crash safety, live autonomous browser execution, and model-free primitives:

📺 Watch on YouTube: https://youtu.be/zAtmsYpyuMY

Chapters: 0:00 The Agent Blind Spot • 0:30 The Fatal moov atom Crash • 1:04 One Context Manager SDK • 1:37 Live Headed Chromium Checkout • 2:11 Four Model-Free Primitives • 2:54 Measured Hackathon Benchmark • 3:28 Local VLM Triage (82.6% Token Cut) • 3:58 Production Ready & MIT License.

• GitHub Repository: https://github.com/0dvsh0/dynamo
• Interactive Demo & Token Calculator: https://0dvsh0.github.io/dynamo/

Who Would Actually Pay for This? (The Side Hustle)

Anyone running autonomous coding agents, QA browser fleets, or computer-use sandboxes in production hits two walls immediately: multimodal inference bills and security compliance.

Dynamo follows an open-core infrastructure side-hustle model built for engineering teams who need visual observability that their security team will actually sign off on:

• Who buys it: AI dev-shop agencies, QA automation teams running parallel Playwright workers on Kubernetes/EKS, and startups building computer-use agents that interact with sensitive staging environments, internal admin panels, or checkout flows.
• Why they pay: SaaS session-replay tools charge per-seat or per-minute and require shipping recordings of proprietary code and customer data to a third-party cloud. Dynamo is a self-hosted library with zero data egress and zero GPU requirement for capture.
• How it monetizes: The core Python SDK and CLI are open-source to drive bottom-up developer adoption. The paid side hustle offers custom sandbox integration, VLM token-budget tuning, and fleet-scale deployment packs (isolated display pools running hundreds of parallel headless sessions per node pool, plus audit-grade compliance artifacts) with direct engineering onboarding.

The Problem: Why DIY Agent Observation Fails

When an agent fails, hand-rolled shell scripts and naive time-based sampling fail in two catastrophic ways:

1. Silent Failure & Corrupted Files on Crash: Standard ffmpeg holds the MP4 container index in memory until a clean shutdown. When an unhandled exception or CI job timeout triggers SIGKILL, the header is never written:

   DIY script, agent killed mid-run : moov atom not found    (0 bytes, 0 frames — unrecoverable)
   Dynamo,     agent killed         : playable MP4, 4/4 logged actions (SIGINT exit path)

1. The 38% Duplicate Token Disaster: Naive time-based sampling captures 1 frame per wall-clock second regardless of screen activity. In our benchmark, 19 of 50 consecutive frames were pixel-identical, burning 56,100 tokens on static screens while waiting on network I/O.

The Solution: One Context Manager

No daemons to babysit, no DISPLAY variables to export manually, and no ffmpeg flags to memorize:

from agent_visual_harness import VisualHarness

# 1. Enter context: Xvfb starts, readiness probe passes, ffmpeg begins recording
with VisualHarness(display=":99", resolution=(1920, 1080), record=True) as harness:
    # 2. Run browser, agent process, or Playwright tests
    harness.run_on_display(["python", "run_agent.py"])

    # 3. Log actions into telemetry.jsonl
    harness.log_action("click", x=500, y=300, target="#pay-button")

    # 4. Instant screenshot bytes mid-run
    screenshot_bytes = harness.take_screenshot()

# 5. Upon exit (even on exception!), review artifacts are exported cleanly:
artifacts = harness.export(
    video_path="output/repro.mp4",
    gif_path="output/demo.gif",
    vlm_keyframes_dir="output/vlm_frames/"
)
print(f"Generated {len(artifacts.keyframes)} frames for VLM context.")

Or run any existing shell-driven agent or test suite directly from the CLI:

# Start a managed Xvfb display + recorder and run any command inside it
$ dynamo record --display :99 --resolution 1920x1080x24 --fps 24 \
    -- python run_agent.py

# Compile PR-ready artifacts with click overlays baked in
$ dynamo export ./dynamo-session \
    --mp4 output/repro.mp4 --crf 23 \
    --gif output/demo.gif --max-colors 128 --dither bayer \
    --overlay-clicks

# Produce a token-budgeted keyframe set + manifest.json for your local VLM
$ dynamo keyframes ./dynamo-session --scene 0.05 --max-size 1280 \
    --format webp --out output/vlm_frames/
# ✓ 29 keyframes · 1,043 frames scanned · manifest.json written

Live Run: Autonomous Headed Chromium Checkout

Inside demo/assets/dynamo_repro.mp4, a headless Chromium browser runs inside Dynamo's virtual display (:99), navigating a multi-step e-commerce storefront:

• 02.0s: Enters customer shipping address.
• 06.4s: Types postal code "97" — Validator Trap: Storefront rejects invalid 2-digit postal code.
• 11.6s: Agent inspects page, corrects postal code to "97209", and clicks Continue.
• 17.1s: Enters credit card details.
• 25.1s: Clicks Pay — Issuer Decline 1: Refused with do_not_honor.
• 31.7s: Clicks Pay — Issuer Decline 2: Refused with do_not_honor.
• 38.3s: Clicks Pay — Authorized! Charge captured on attempt 3 ($429.56).
• 42.0s: Order Confirmed (#NW-9821) with click overlays timestamped into telemetry.

Four Model-Free Arithmetic Primitives

Recording gets an agent a replay. Four more primitives get it an answer: what error the page showed, which pixels moved, where the button is, and which coordinate space those pixels are in.

These four are pure arithmetic — no model involved, which is why they are fast, reproducible, and free:

# 1. Capture a specific region, not the whole screen (crops in ffmpeg before encoding)
$ dynamo capture --region 0,0,480,90 --out banner.png

# 2. Text off pixels — for canvas, VNC, RDP, and BIOS surfaces that give up no DOM text
$ dynamo ocr banner.png --find DO_NOT_HONOR --fuzzy   # exits 1 if absent

# 3. Visual regression gate — exits 1 when the UI moved
$ dynamo diff before.png after.png --out diff.png

# 4. Number elements so a model answers "12" instead of guessing "(740, 318)"
$ dynamo som shot.png --boxes boxes.json --labels Pay,Cancel --out som.png --resolve 12

Visual Inspection Gallery

1. Region Snip vs. Full Viewport (20× Byte Cut)

Cropping before encoding shrinks a 92 KB full screenshot down to a 4 KB snip. Vision models pricing by image dimensions or bandwidth receive only the error banner:

Region Snip (4 KB)	Full Viewport (92 KB)

2. Visual Regression Diff Overlay

Highlights moving UI elements with a red-on-dimmed bounding box overlay. A control comparison with unchanged UI reports zero changed pixels:

3. Set-of-Marks Numbered Element Grounding

Numbers elements directly on the live framebuffer so VLMs answer "12" rather than guessing floating-point coordinates:

4. Pixel OCR on Non-DOM Surfaces

Extracts text directly from <canvas>, VNC, or RDP frames with fuzzy confusable matching (D0_ $\rightarrow$ DO_):

What the Benchmark Actually Shows

We evaluated the exact same checkout workflow captured three ways — plain per-action screenshots, DIY wall-clock ffmpeg, and Dynamo:

The honest takeaways:

• Dynamo and screenshots cost similar tokens (31,900 vs 26,400) because both sample on actions rather than wall-clock time. Dynamo's win is everything screenshots cannot do: the video, the GIF, click overlays, timing manifest, crash resilience, and a 2.6× smaller payload using WebP.
• Time-driven sampling is the real token disaster. The DIY script kept 1 frame/sec and paid 2.1× tokens with 19 of 50 frames pixel-identical.

Intelligent VLM Triage: Estimated vs. Counted

For questions arithmetic cannot answer — what did the decline message say? — Dynamo routes frames through a 3-stage funnel:

Key findings:

1. Cropping saved almost nothing in model tokens (0.3%). Because local VLM servers resize incoming images to a fixed internal token grid, a full frame and a small crop cost the exact same tokens. The crop cuts wire bytes and points the model at what moved, but is not a token saving here.
2. Skipping frames via a cheap decision model (laya) did the real work. In Mode C, a lightweight typed decision model (Laya) scored candidate frames and dropped 5 low-signal frames before they ever touched the vision encoder. Not sending the frame cut prompt tokens to 82.6% and reduced wall time from 255s to 83s.
3. 100% Local & Offline: Powered by Apache-2.0 models (qwen3-vl:4b and laya), running entirely on CPU/GPU without cloud API costs or data egress.

Prize Categories

• Overall Winner (Side Hustle): Dynamo solves an expensive infrastructure problem for teams building autonomous AI agents, backed by transparent benchmarks, a zero-dependency Python SDK/CLI, model-free arithmetic inspection, and a clear B2B open-core monetization path.

• Best Use of GitHub Copilot / GitHub Actions Integration: Built for native GitHub Actions CI workflows (dynamo record -- pytest tests/e2e $\rightarrow$ dynamo export --gif pr/demo.gif --overlay-clicks) so every pull request automatically receives a visual replay artifact.

• Best Use of Sentry Agent Tracing / Agent Observability Spirit: Delivers full visual + JSONL action telemetry (telemetry.jsonl), four model-free inspection primitives, and side-by-side estimated vs. actual model token accounting.