JVT:以簡潔的大型 patch Transformer 簡化像素空間影片擴散
JVT: Just Video Transformer makes pixel-space video diffusion simple
JVT(Just Video Transformer)以簡潔的大型 patch DiT 式 Transformer 直接進行像素空間影片擴散,無需預訓練自動編碼器或 U-Net 跳接,生成品質媲美潛在擴散基線。JVT 亦支援相機條件控制的 RGB-視差聯合生成,相較只生成 RGB 的基線改善相機跟隨,並可在推理時毋須額外重建模型,產生明確的場景幾何。
Published on Nov 13
Authors:
,
Abstract
JVT, a simple large-patch DiT-style Transformer, enables pixel-space video diffusion without pretrained autoencoders or U-Net skip connections, achieving competitive quality and supporting joint camera-conditioned RGB-disparity generation.
Direct diffusion modeling of high-dimensional raw video data is challenging, and existing pixel-space methods often rely on specialized architectures. We introduce JVT (Just Video Transformer), a simple large-patch DiT-style Transformer for pixel-space video diffusion, without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net-style skip connections. We systematically examine video patch embeddings, register tokens, and adaptive layer normalization to identify design choices that balance generation quality and computational efficiency. The resulting JVT design achieves generation quality competitive with latent diffusion baselines in both class-conditioned and unconditional settings. The pixel-space formulation of JVT also naturally supports joint generation of pixel-aligned modalities. We demonstrate this capability through camera-conditioned RGB-disparity generation, which improves camera following relative to RGB-only baselines and yields explicit scene geometry without an additional reconstruction model at inference. These results highlight the potential of simple DiT-style Transformers for pixel-space video diffusion, and we hope our findings offer useful insights and practical guidance for future research.
View arXiv page View PDF Project page GitHub 4 Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.18549 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.18549 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.18549 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co