Blazar：面向生成式影片的影片編碼器

Lately I've been using video diffusion models and learning more about them. As I was digging around the various codebases I noticed that there were a couple of common approaches to actually encoding the frames that are generated by the diffusion models: mainly use PyAV or shell out to FFmpeg. There's nothing especially wrong with either of these approaches but I think it is possible for us to do a little better. So I (along with my robot friends) built Blazar. You can find the code repo for Blazar here.

Today with PyAV the flow is roughly:

1. Convert tensor from BF16 or F32 to u8 and reshape to (H, W, C)
2. D2H transfer
3. RGB to YUV conversion
4. (if NVENC) upload to the GPU
5. Encode

So, I think, this is roughly what we want to do:

1. Be able to take the native output tensor shape of the model as an input. (B, C, T, H, W) seems to be the common tensor shape but a video encoder will want more like (B, T, H, W, C) or (B, T, C, H, W) (or whatever nv12 is with U' and V' interleaved)
2. Do a fused YUV + tensor reshape conversion (does that count as a reshape? maybe not) either on GPU or CPU
3. Encode either using NVENC (or some other HW encoder) or a variety of software encoders
4. Be relatively easily configurable...
5. Probably don't get in the way of the Python GIL

So the robot and I landed on a design where most of the work happens in Rust. Backends for YUV conversion and encoding are swappable between hardware and software so that if you would prefer not to use HW for YUV conversion and encoding you don't have to do that. There is a concept of "Calibrations" that are keyed on model and target quality so you can have multiple calibrations with different latency or quality targets and make them configurable by model. These live in human readable TOML files.

import blazar

cals = blazar.Calibrations.load("calibrations/default")
with blazar.Encoder(calibrations=cals, identity={"family": "wan", "version": "2.1",
                                                 "quant": "bf16", "size": "1.3b"},
                    target="balanced", backend="nvenc", codec="h264",
                    output="out.mp4") as stream:
    stream.submit(frame, width=W, height=H, fps=24.0, layout="thwc", range="unit01")
    # or, with device tensors:
    stream.submit(tensor, fps=24.0, range="unit01")

Here's a pretty simple example of using Blazar to encode frames. You can submit them one at a time or as a chunk of video, in which case they'll be mass converted to YUV and then encoded.

Peformance Comparison

This comparison table was done on a Core i7 14700KF with a 4080 Super in WSL. What we are measuring here is a BF16 (1, 3, 121, 1088, 1920) tensor being passed into the encoder pipelines for these scenarios:

1. Blazar device bf16 (BF16 input tensor, GPU YUV conversion, NVENC)
2. Blazar device u8 -> CPU YUV Conversion -> NVENC
3. PyAV device u8 -> CPU YUV Conversion -> NVENC
4. FFmpeg device u8 -> IPC -> CPU YUV Conversion -> NVENC

The optimal path is the device bf16 path. Here we feed the tensor directly into a CUDA kernel that simultaneously converts from RGB to YUV and reformats the buffer into the shape NVENC expects (T,C,H,W). On my 4080 Super we're able to hit 258 fps with 1088p tensors from SGLang. If we copy to host as a u8 that drops to 158 fps. Compared to the PyAV or FFmpeg CLI paths this is a fair improvement still.

Looking at how Blazar scales with thread count you can see we 4x the throughput with 4x the threads. Really this test was to try to induce some contention in Python's GIL (Global Interpreter Lock) to test its impact on PyAV. You can see it basically caps out at 2x regardless of the number of threads.

The threads here are independent encodes, not breaking individual encodes up into threads which can improve performance at the potential cost of image quality.

Where we're at today

Today the project has support for:

• Hardware encoders: NVENC (AV1, HEVC, AVC), VideoToolbox (HEVC, AVC)
• Software encoders: x264, x265, vpx (VP9), svt-av1
• Opus audio codec is currently supported
• YUV conversion on GPU or CPU
• fMP4 or Buffer output
• Custom configurations

Where I want to take this

I have ideas for the future of the project. I think if we start to see some adoption there will of course be bugs and feature requests. I think that having model-aware and scene-aware encoding might be an interesting feature. I also want to explore using latents to identify regions of interest or use other signals to inform encoding decisions.

Anyway if you are running generative video models please check this project out, I am happy to take any feedback! If you want to contribute that is even better!