ClearPix 瀏覽器影片升頻支援 4K，毋須上傳且靜態片段快 4.7–7 倍

Two things were true about our video upscaler when this round started: it was honest about being slow, and we had no idea whether it was any good. The first half is fixable on paper. The second half is why we spent a week building a ruler before touching a single line of the optimizer. What came out the other side: static shots now upscale 4.7–7× faster, talking-head footage — a static background with a moving subject, the case that used to get zero benefit — goes from a flat 1.00× to about 1.8×, and the output is measurably sharper than what we shipped last month. All of it inside a browser tab, with your video never leaving your machine. Not "we delete the file after processing": there is no upload path in the code.

We build ClearPix, the in-browser media toolbox from parts 1–10 of this series — no uploads, no server GPUs, Real-ESRGAN-class super-resolution running on WebGPU inside your browser. The video pipeline itself (hardware decode, per-frame AI, hardware encode, audio passthrough, O(1) memory) was part 2's story. This round was about turning that pipeline up to 4K and answering the two questions that actually matter for a video tool: how fast, and how good?

Act 1: Build the ruler before you touch the dial

We have a house rule by now, and we earned it the expensive way: no optimization without a per-stage timing breakdown, and no quality change without a metric that can veto it.

The video path had the first half. bench-video-upscale.mjs reports per-frame milliseconds across every output tier, with the decode/inference/encode/paste split, on real clips. What it did not have was a quality number. Its outputPsnr measures the deviation between "frame reuse on" and "frame reuse off" — a regression guard for a specific optimization, not a statement about how the picture compares to the truth. So "is our video upscaling any good?" was, scientifically speaking, a vibe.

The fix was a ground-truth bench (bench-video-quality.mjs). We take our natural test photos at 800×500 as ground truth, synthesize a low-resolution source by downsampling each one by a factor (f = 2, 4, 8), run the exact production geometry on it, bring the result back to ground-truth size, and score PSNR, SSIM, and high-frequency variance against the original. Same seed, same model, same tiles — only the geometry changes between rows.

One reading caveat, stated up front, because it nearly fooled us: Real-ESRGAN is a perceptual GAN. PSNR actively under-rates it against a dumb bicubic resize (at f = 4 on a clean source: 23.57 dB versus bicubic's 25.13 dB), so these numbers must never be used to argue "AI versus bicubic." They are only valid for model-against-itself comparisons — and there, they are exactly what we needed. Two independent metrics (PSNR and SSIM) agreeing in direction is our bar for believing a result.

The ruler immediately found something embarrassing. Our geometry was resizing the source down to a quarter of the target — 960px wide for a 4K output — and handing that to the model. Any real detail living between that quarter-target resolution and the source's actual resolution was being thrown away before the AI ever saw it. We had been carefully building a pipeline that discarded the user's own pixels.

Act 2: The dial nobody had turned

Feeding the model a bigger input means it does its ×4 on a larger frame, and we then scale the result back down to the target. That sounds like extra work for nothing — the model was already being asked to upscale. It is not nothing: at the larger input, the source's genuine detail is still present, the ×4 invents detail around it, and the average of that dense 4× field down to target size quietly smooths out the model's own artifacts.

We swept the input scale (m = 1.0 is what we shipped before) across all four images and all three downsampling factors — twelve rows, averaged:

Two rows killed their own ideas, and one needs a warning label. Start with the last row: "just feed the model the source at its own width" looks like a better trade than m = 1.5 — more gain for less cost — but it isn't a setting, it's a relationship. The input scale that equals your source's own width changes with every source-to-target ratio (2 at f = 2, 1 at f = 4, 0.5 at f = 8), and at the small end it degenerates into the m = 0.5 we are about to bury. We tested the intuition behind it directly — "stop pre-upsampling past the source" — and at f = 8 the geometry we ship still beat it, by 0.6–0.8 dB on three of the four test images (and gave back 0.3 dB on the fourth). Pre-upsampling is not a bug we forgot to remove.

The top row buries a piece of folklore we had been carrying around for months: "feed the model less resolution and let it work harder" costs 1.88 dB on average. That direction is dead, and we wrote it into the runbook so nobody re-proposes it in six months.

m = 1.5 was the row that shipped, for a boring reason: it was non-negative on every single one of the twelve rows — +1.88 dB at f = 2, +0.77 dB at f = 4, +0.14 dB at f = 8. An optimization that can't go backwards is an optimization you can hand to users as a choice. Note the cost column: it follows the area of the AI input (1.5² = 2.25 nominal, 2² = 4, 3² = 9) rather than the tile count, and every number in it is measured, not assumed.

So the option became a three-position dial. Fast is the old geometry. Balanced is m = 1.5. Best is m = 2, where the curve starts bending — another +0.7 dB for nearly double the time. We also stopped asking users to guess what their machine can afford. The default position comes from device capability plus the clip in front of you. No WebGPU — where the fallback WASM path costs roughly 25× per frame — or a low-end device, and you land on Fast automatically. Otherwise we estimate each tier's finish time from the file you actually dropped: its duration and frame budget, with the real fps probed when the estimate is close enough to matter. The most expensive tier that still fits its time budget wins. A six-second 720p clip on a laptop with a real GPU defaults to Best. A four-minute 4K export defaults to Fast, without anyone having to be told why, and the number next to the buttons updates the moment you change the target. It is not a marketing figure; it is arithmetic, and it happens to be right.

Act 3: Teaching the upscaler to skip the boring parts

Here is the uncomfortable shape of the speed problem: we profiled everything, and inference is 97% of per-frame time. Decoding is nearly free (the OS media engine does it on silicon). Encoding is nearly free. Pasting tiles to a canvas is noise. There is exactly one lever, and it is not "make inference faster" — it is "don't run inference". Video is full of frames that did not change.

We already had frame-level reuse for this: compare each frame against the last one we actually ran through the model, and if it hasn't moved, reuse the previous output. The implementation detail that cost us real time is worth repeating because it is counterintuitive: the comparison cannot be byte equality. Decoded H.264 static frames are not bit-identical — we measured a max delta around 9 and a mean of 0.02 between consecutive "identical" frames, because the encoder makes slightly different decisions per frame. Byte comparison scores 0% hits on video that is visually frozen. The fix is an anchored, bounded comparison: each frame is compared to the last frame that really went through inference (not to its predecessor, so deviation can't chain), and a hit reuses that anchor's output without moving the anchor. Bounded tolerance, no drift. This is the same discipline we landed on for video watermark removal in part 7.

The numbers on real clips at τ = 1.0:

And then the obvious follow-up question, which took us embarrassingly long to ask: what does real footage actually look like? Talking heads. Screen recordings with a cursor. Product close-ups. All of them are subject-motion clips — a static background with something moving in the middle — and every one of them gets a 0% hit rate at frame level. The most common shape of video on the internet was the shape our optimization couldn't touch. One moving person flips the entire frame back to full inference, forever.

So we pushed the same idea down a level. The frame is already processed in 256px tiles internally, so each tile now keeps its own anchor and gets its own bounded comparison. A tile that hasn't changed copies its core region straight from the previous output canvas — no format conversion, no inference, no paste — while the tiles that did change go through the model as usual. Anchors still don't update on a hit, so the drift bound is the same as the frame-level version.

Measured in isolation on synthetic clips (24 frames, 1080p tier = 4 tiles), the results split cleanly:

The last row is the honesty check: when nothing is reusable, tile-level costs about 1% (one comparison per tile) and correctly returns nothing. The middle row is the whole point — the 1.79× comes with a deviation of 49.77 dB against no-reuse output, which is a fraction of one 8-bit level. Invisible. And since frame-level is strictly better when the whole frame is static, we ship them stacked: frame first, tiles as the fallback layer. Users never choose; the picture decides.

One trap almost shipped, and it's a good argument for building the bench before the feature. Pasting a tile back has to use the core region — the 8-pixel overlap around each tile exists only to give the model context, and must not be written to the canvas. putImageData's dirty rectangle is relative to the source ImageData, not to the destination, so the destination offset has to be the tile origin; pass the core origin instead and every tile lands 32 pixels off. Our PSNR gate caught the shift: 54 dB became 16.9 dB. Visually, "32px of double-exposed edge" reads as slightly soft — exactly the kind of thing that ships and gets discovered by a user a month later.

Why the privacy part is structural, not a policy

Everything above runs in your tab: hardware decode via WebCodecs, inference via ONNX Runtime Web on WebGPU, muxing via MediaBunny, hardware encode back to H.264. There is no server component of the video path at all. The only bytes that travel are the model weights coming down — once, then cached in IndexedDB, after which the tool works offline — plus a handful of cookieless analytics events: an event name and coarse properties like backend and elapsed seconds, the full list published on our privacy page. No media, no filenames, no durations tied to a person.

That is a stronger claim than "we don't store your videos," and it is checkable from the outside: open devtools, drop a clip in, watch the network tab. Models come down; nothing goes up. This is the part that gets described as a marketing position by people who haven't tried the alternative, and it genuinely costs us — a server GPU would make the long-clip problem evaporate, and a queue would let us serve 4K to a five-year-old phone. It would also mean your footage crossing the wire to be upscaled, and that is the one trade we don't make. If you're upscaling something you wouldn't email to a stranger — a family video, an unreleased product shot, a client's footage under NDA — the tool's whole job is running on the laptop you already trust.

Where the walls are

Per-frame cost on our reference machine (Apple GPU, WebGPU, moving content): 720p ≈ 115 ms, 1080p ≈ 263 ms, 2K ≈ 464 ms, 4K ≈ 1.06 s. Inference is 97% of each of those numbers, which is why this post is about doing less inference rather than doing inference faster.

We also removed the hard duration cap this round. There's now a size guard (200 MB) and a frame budget, an honest per-clip estimate shown up front, and if the estimate exceeds eight minutes we ask instead of refusing. Slow is a conversation now, not an error message.

Dead ends, recorded so we don't walk them again: multi-worker parallelism is a no-op (inference is memory-bandwidth-bound; more workers only fight over the bus), bigger tiles are a wash, IOBinding buys 1–2% so we skipped it, feeding the model less resolution costs 1.88 dB, and detail injection at k = 0.7 helps clean sources but re-amplifies compression noise on real ones, so it stays on the bench with no perceptual metric to justify it. One more for the file: if you benchmark two variants back to back, the first one silently absorbs the video encoder's cold start and looks 2.17× faster for free. Warm up first or you will ship the wrong winner.

Try it

No account, no upload, no watermark on the output: upscale a video to 4K in your browser — the tier dial and the estimate are under the target selector, and the frame budget will tell you the truth before you commit. The in-browser image upscaler shares the model and the tile engine, the free video watermark remover shares this exact pipeline, and if the transport layer is the part you're here for, part 2 is the read.

The nine seconds and the plastic skin were the same complaint. This round's version of that: "fast" and "sharp" and "private" were never three separate problems — they were one pipeline, observed from three angles.

Part 11 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.