Screen2Tutorial：把編碼及終端機錄影片段轉成逐步 Markdown 教學

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Screen2Tutorial turns a short screen recording of a coding or terminal tutorial into a step-by-step written guide in Markdown, with the commands and code copied from what appeared on screen.

If you make coding videos, you know the chore of writing the blog-post version: pause, rewind, squint at the terminal, retype each command, and hope you didn't mistype one. Screen2Tutorial does that first draft for you. It is aimed at independent coding educators, YouTubers and developer-relations people who publish a video and a written version of the same lesson.

The part I care most about is trust. A model that writes tutorials will sometimes invent a command or change an argument. So every command it writes is checked against two sources: the spoken transcript and the text read from the video frames. Each command ends up marked as verified by the screen, by speech, by both, or UNVERIFIED, so the creator knows which lines to re-check before publishing.

Would someone pay for it? That's a hypothesis I haven't validated. My guess is that a creator would pay to save editing time, as long as they can see where each step came from.

Scope, on purpose: one English-speaking narrator, terminal or IDE recordings, short videos, Markdown output. No logins, payments or multiple languages.

Demo

This is the link of my video demo (link)

I tested it on two public videos. The main one is "10 Basic Ubuntu Linux Commands Every Beginner Should Know" by vCloudBitsBytes (link), about 4 minutes long. Here is one step of the output, with the verification label next to each command:

Step 4: Create Directories and Files using MKDIR and TOUCH
The author demonstrates how to use the mkdir command to create a new directory named test and the touch command to create a new file named test.txt...

mkdir test — verified_screen
ls — verified_both
touch test.txt — verified_screen

Code

screen2tutorial

Turn a 5–10 minute English coding recording into a code-backed, step-by-step Markdown tutorial. Speech transcription and screen OCR are aligned by time The writing model organizes the explanation and extracts visible code examples each shell command is compared with both evidence sources, and commands found in neither are marked UNVERIFIED.

The supported scope is one English-speaking narrator, one terminal/IDE recording, and Markdown output. The application has a Streamlit upload page and the CLI python -m pipeline.run path/to/video.mp4.

Example output

Excerpt from a generated guide for a CSS random() tutorial:

The author demonstrates how to generate random values directly within CSS to avoid hard-coding manual values or using JavaScript for simple visual effects.

particle {
  scale: random(1px, 7px);
}

Generated code is included when it matches text recognized on screen. Review the extracted snippet against the recording before relying on it.

…

MIT licensed. The README has Windows install steps (including ffmpeg and Tesseract), configuration, limitations and tests.

How I Built It

The pipeline has six stages, one module each:

1. ffmpeg extracts the audio and a frame every 2 seconds. Frames that are nearly identical to the previous one are dropped, since a screen recording is mostly unchanged screen and this saves a lot of OCR time.
2. faster-whisper (base model, int8, CPU) transcribes the audio with timestamps.
3. Tesseract reads the text from the remaining frames.
4. Alignment groups speech and screen text into 30-second windows, so each model call sees what was said and what was visible at the same moment.
5. The writing model turns each window into one tutorial step as strict JSON. The prompt tells it to include a command only if it appears in the transcript or screen text, and code blocks that don't resemble the OCR text are dropped. Invalid JSON is retried once, then replaced by a plain fallback step.
6. Verification normalizes each command and checks it against the transcript and screen text, using an exact match first and then a fuzzy match with RapidFuzz.

The open AI: the writing step uses Gemma 4, an open-weight model, reached through Ollama's OpenAI-compatible endpoint. Whisper and Tesseract run entirely on my laptop. The endpoint and model name are two lines in config.yaml, so another open model, hosted or local, can be swapped in without touching the code.

Built for a weak laptop. My machine has 8 GB of RAM, an Intel i3 and no GPU. That decided most of the design: a small Whisper model, frame deduplication before OCR, caching of every stage on disk, and a hosted open model for the one step that needs real language quality.

What testing showed

I'm a beginner with Whisper, OCR and LLM APIs, so I built and tested one stage at a time. The terminal test taught me the most.

First run on the Ubuntu video: 20 commands found, 14 verified, 6 flagged UNVERIFIED. When I looked closer, four of the six flagged commands were verified in the neighboring step. The cause was in my own code: each command was checked only against the text of its own 30-second window, so a command typed near a window boundary was checked against the wrong text.

The fix: verify each step against the previous, current and next window together, and require whole-word matches for very short commands like ls and cd (otherwise "ls" matches inside ordinary words like "also").

All six earlier flags were false alarms from the window boundary, not real misses. All ten commands named in the video's own chapter list (cd, ls, pwd, mkdir, touch, cp, mv, rm, sudo, apt-get) appear in the tutorial.

The CSS video (a tutorial on CSS random(), (link)) produced an 8-step guide with 10 code blocks. Its verification summary showed 0 commands, which is correct, because the video was about CSS and not shell commands. It also shows the OCR limits: Tesseract read 0deg as @deg and :root as sroot, and some closing braces were lost. Each code block carries a "check unclear characters against the recording" note for this reason.

Where it is still weak

A 20/20 score on one video is not proof that the verifier catches mistakes, so here are the honest limits:

• "Verified" means "resembles text from the video", not "is correct or safe to run."
• One action can be split across two windows, which produced repeated steps (the touch, rm and apt-get steps each appear twice in the Ubuntu output).
• OCR is imperfect on small or low-contrast text, and I have only tested two videos.

Why Does Open Innovation Matter?

1. Privacy is a setting, not a promise. Tutorials often show half-built projects, internal tools or terminal history. With open weights, the same pipeline can point at a local model for recordings that must not leave the machine. Right now my config uses a cloud-hosted Gemma, which sends the transcript and OCR text to Ollama for the writing step, and the README says so plainly. Switching to a local model is a config change.

2. No lock-in. The model is a config value. If a better or cheaper open model appears, or one disappears, I change a line instead of rewriting the app. A creator who builds a workflow around the tool needs that stability.

3. I could see why it was wrong. The verification feature is only worth trusting if its parts can be inspected. Because the whole stack is open, readable code and components, I could trace the six false alarms to a single design decision, fix it, and measure the difference, which is the loop in the testing section above.