Code2Video-Compass:涵蓋 T2V、I2V 和 V2V 的通用程式碼影片生成綜合基準
Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V
Code2Video-Compass 是一項評估多模態語言模型以程式碼生成影片的基準,涵蓋 T2V、I2V 和 V2V。基準包含 2,104 項任務,涵蓋 10 個類別、24 個子類別,並以 12 項客觀指標和五項主觀指標評估表現。對來自九個模型系列的 15 個專有及開放權重模型進行評估後,研究發現模型在複雜動態及狀態依賴任務上的表現仍面臨挑戰,且任務複雜度提高時表現普遍下降。
Abstract
Code2Video-Compass is a benchmark evaluating multimodal language models on code-based video generation across text, image, and video inputs, revealing significant challenges in complex dynamic and state-dependent tasks.
Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.
View arXiv page View PDF Project page GitHub 1 Add to collection
Get this paper in your agent:
hf papers read 2610.28416
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.28416 in a model README.md to link it from this page.
Datasets citing this paper 1
code2video-compass/Code2Video-Compass
Viewer • Updated about 21 hours ago • 2.1k • 36 • 4
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.28416 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co