使用 WebGL 2 和 Sobel 在瀏覽器將影片轉為線稿

Depth maps and line art preserve different parts of a video. Depth describes spatial relationships; contours emphasize shape, composition, and visible changes in pose.

After adding depth-video conversion to Timeline Studio, my open-source browser video editor, I added a line-art asset generator. This version runs locally with WebGL 2 and Sobel edge extraction. It does not download or run a neural line-art model.

Source code · Try the editor

Here is how the image-processing pipeline connects to actual video editing: previews, trimmed clips, speed curves, source audio, encoding, and cancellation.

Two passes on the GPU

The pipeline is deliberately small:

Image or decoded video frame
          |
   Grayscale + 3x3 smoothing
          |
   Intermediate GPU texture
          |
   Sobel + soft threshold
          |
    Output canvas
          |
      PNG or WebM

The smoothing pass renders into a texture attached to a framebuffer. The edge pass samples that texture and draws to the output canvas. Intermediate pixels stay on the GPU rather than being read back into JavaScript between passes.

Shader programs, buffers, textures, and the framebuffer are reused across frames. That reduces setup work, although decoding and encoding still contribute to total processing time.

Convert to luminance, then smooth

The shader computes a weighted RGB luminance:

float gray(vec2 p) {
  return dot(
    texture(source, p).rgb,
    vec3(0.2126, 0.7152, 0.0722)
  );
}

The first pass uses this weighted 3x3 kernel, divided by 16:

1  2  1
2  4  2
1  2  1

Smoothing suppresses some high-frequency texture before edge extraction. It is a tradeoff: small details can also become weaker.

Because detection operates on luminance, different colors with similar luminance may produce weak boundaries. Conversely, a shadow or reflection can produce a strong edge without marking an object boundary.

Extract Sobel gradients

The second pass estimates horizontal and vertical gradients:

Gx                  Gy
-1  0  1            -1 -2 -1
-2  0  2             0  0  0
-1  0  1             1  2  1

Their combined magnitude becomes the edge strength:

float gradient = length(vec2(
  c + 2.0 * f + i - a - 2.0 * e - g,
  g + 2.0 * h + i - a - 2.0 * b - c
));

The letters represent neighboring luminance samples. Each output pixel performs the same local calculation, which maps naturally to a fragment shader.

Make the parameters useful

The inspector exposes detail, line weight, and inversion. Defaults are 55% detail, 1x line weight, and black lines on white.

Detail maps to an edge threshold:

float threshold = mix(0.42, 0.035, detail / 100.0);
float ink = smoothstep(threshold * 0.5, threshold, gradient);
float value = inverted ? ink : 1.0 - ink;

Higher detail lowers the threshold, retaining weaker edges. The soft threshold avoids an abrupt binary transition around the cutoff. For busy footage, increasing detail can make the result less useful by retaining background texture.

Line weight changes the sampling distance. It uses the longest dimension as a 1080px reference:

const scaledThickness =
  thickness * Math.max(width, height) / 1080;

This helps keep the edge-sampling scale comparable across output sizes. It is not morphological dilation: changing the sampling distance also changes which details respond. The initial blur itself remains a one-pixel 3x3 neighborhood.

Preview and asset generation share one renderer

Both paths call the same implementation:

const renderer = createLineArtRenderer(canvas, width, height);
renderer.draw(media, options);
// Release resources when finished:
renderer.dispose();

The preview redraws a sampled frame when parameters or the local playhead time change. Asset generation traverses output frames through that same renderer.

Sharing the implementation reduces differences in thresholds, sampling, and inversion between preview and generated assets. It does not make processing instantaneous.

Output dimensions preserve aspect ratio, cap the longest edge at 1080 by default, avoid enlarging smaller media, and normalize dimensions to even numbers.

Video has to follow the timeline's clock

A selected clip may already be trimmed or retimed. Sampling the original file from zero would produce the wrong result.

There are separate coordinates for output time, clip-local time, and source-media time. For constant speed:

sourceTime = sourceIn + clipLocalTime * speed

For speed curves, the implementation reuses the editor's mapping:

getVisualSourceTime(segment, localTime)

The generator outputs 24 fps. Each local sample time is i / 24; it is mapped to source time before decoding the frame. The processed frame is then encoded at its output timestamp.

Output frame time
       |
Source-time mapping, including trim and retiming
       |
Decode source frame -> draw contours -> encode output

Mediabunny handles media input and timestamp-based canvas sampling. The final frame duration is limited to the remaining clip duration.

24 fps is the output rate, not a claim of real-time generation speed. Actual throughput depends on the browser, hardware, source media, and encoder.

Normalize still-image orientation

One integration detail had nothing to do with Sobel: image orientation.

The image path decodes a Blob to an ImageBitmap, then draws it to a 2D canvas before uploading it to WebGL. ImageBitmap does not follow the same UNPACK_FLIP_Y_WEBGL behavior as the other upload path, so the intermediate canvas normalizes orientation.

Without that step, still images and video frames could disagree about which side is up.

Encode a reusable asset, with corresponding source audio

Images become PNG. Videos become WebM using VP9 at approximately 4 Mbps. When corresponding source audio exists, the output uses Opus at 192 kbps.

Audio preparation reuses the editor's source-audio and offline-mixing helpers so trimming and retiming follow the selected segment. This preserves the segment's corresponding source sound; it does not mix every music and narration track from the whole project.

The completed asset goes to My assets. It is not automatically inserted into the timeline, making it possible to retain the original and compare versions before placement.

Cancellation includes cleanup

An AbortController carries cancellation through source reading, frame processing, and output finalization. The renderer releases its textures, framebuffer, shaders, programs, and vertex buffer. Media input resources are disposed, and encoding is canceled when necessary.

Frame processing reports progress up to 99%. Finalization and the asset commit must finish before the UI reports completion. Processing the last frame is not the same as having a usable saved file.

What structural reference means here

Depth video and contour video provide complementary reference assets:

A downstream generation workflow can use these only if it accepts the corresponding conditioning input and its preprocessing, timing, resolution, and frame-rate requirements are satisfied.

This feature creates the reference asset. It does not imply automatic integration with every image or video generator, nor guarantee that a generator will preserve the desired structure.

The current Sobel route also lacks semantic simplification and explicit temporal tracking. Fine textures, noise, and motion can make lines fluctuate across frames. A learned line-art model or additional temporal processing would need separate evaluation rather than being claimed as part of this implementation.

Try it or read the code

Open Effects -> Line art, select an image or video clip, adjust parameters, and generate the asset. The feature includes 13 interface languages and a mobile layout.

Implementation entry points:

• src/lib/lineArt.js: renderer, source sampling, encoding, and cleanup
• src/components/LineArtPanel.jsx: preview, controls, and job state
• src/i18nLineArt.js: localized copy

If you work on browser graphics or video tools, I would welcome difficult examples: busy backgrounds, small contours, and fast motion are particularly useful for understanding where this approach needs improvement.

GitHub repository · Open the editor

Disclosure: I maintain the project discussed here. This article was drafted by an AI assistant using the current repository source and implementation notes.