Shotcat 如何在瀏覽器中從螢幕錄影找出點擊位置

I wanted a phone recording to show where the finger was. A plain screen recording has no touch data: you get a video of a screen reacting to something invisible.

So I tried to recover the touches from the pixels. The video stays in the browser, and only taps that need a judgment call go to a model, as two small stills.

Step 1: find when the screen reacts

Decoding is cheap at low resolution. I decode at 12 fps to 72 px wide grayscale with mediabunny on top of WebCodecs, then compare neighboring frames:

function changedShare(a: Float32Array, b: Float32Array, w: number, y0: number, y1: number) {
    let n = 0;
    for (let y = y0; y < y1; y++)
        for (let x = 0; x < w; x++)
            if (Math.abs(a[y * w + x] - b[y * w + x]) > PIXEL_DELTA) n++;
    return n / ((y1 - y0) * w);
}

A pixel counts as changed when it moves by more than 18 out of 255, and a frame counts as changed when more than 0.4% of its pixels do. The top 7% and bottom 3% are ignored, because the clock and home indicator change on their own.

Consecutive changed frames form a run. The first version merged anything close together, and that lost taps made right after a scroll. Now a sharp rise while the previous change is fading starts a new run.

Step 2: tell a swipe from a tap

12 fps is too coarse here: a fling moves content more than a quarter of the screen between frames. So each run is decoded again at 48 fps, and every frame step is tested for motion. I shift one frame against the other and look for the offset that explains the difference best:

• vertically across the whole screen, for scrolling and sheets
• horizontally inside the band that changed, because a carousel moves only its own row

If shifting removes at least 40% of the difference, that step is motion. The timing then settles the question. A swipe moves content at once and keeps it moving one way. A tap changes the screen first and animates afterwards.

Movement within half a second after a swipe is momentum or bounce, so it's skipped. Swipes are placed from the measured motion alone, with no model. Flings fly much further than a finger travels, so the drawn path is capped.

Step 3: ask where the finger was

A changed region isn't the tapped control. Tap a row and the whole screen slides away. This is where a vision model earns its place.

For each tap the browser sends a 768 px JPEG pair, before and after, plus the box of the first pixels to change. That box is a strong hint: a small one is usually the pressed control lighting up, and a full-screen one means navigation. The prompt asks which control in BEFORE leads to AFTER, and it allows tap=false for timers, spinners and video.

Each moment is its own request, run side by side. The finger goes to the center of the returned box, 0.15 s before the screen reacted, because the app answers a beat after the finger lifts.

Which model

I tested on one real 8 second recording with 5 taps:

One recording isn't a benchmark, so treat this as a hint for your own test. Thinking is off, temperature is 0.1, and one tap costs about 2.4k tokens, which came to roughly $0.02 for 10 taps at the price when I tested.

What's still weak

• Typing. A keyboard produces a burst of small changes, and the model is told to skip it rather than draw a run of key taps.
• Taps that change nothing on screen are invisible by design.
• Swipe paths are estimates. The start point comes from a heuristic, since the content's motion doesn't say where the finger was.

What I'd keep

The split worked better than sending the whole video to a model. Cheap pixel math answers "when" and "what kind", and the model only answers "where". That keeps the recording on the user's machine, keeps the cost small, and gives the model an easy question.

This is how taps work in Shotcat, the tool I'm building.

Drafted with AI assistance from my code and test notes; I reviewed the facts and numbers before publishing.