跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 24 天前AI 評分42

WorldFlow:透過大規模預訓練聯合預測影片與 3D 動作

WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

AI 導讀

WorldFlow 是生成式世界模型,透過連續位移 token 與 Mixture-of-Transformers(MoT)骨幹,聯合預測未來影片和 3D 軌跡。在 SpatialCog 評測集上,其 ADE 為 0.060 m,較最強外部基線降低 54.2%;相較僅使用 flow 的變體,ADE 和 FDE 分別降低 55.9% 和 68.5%。

正文

Published on Oct 29

Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

WorldFlow is a generative world model that jointly predicts future video and 3D trajectories using continuous displacement tokens and a Mixture-of-Transformers backbone, improving physical dynamics prediction across diverse benchmarks.

Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce , a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2\% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9\% and 68.5\%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.

View arXiv page View PDF Project page GitHub 17 Add to collection

Get this paper in your agent:

hf papers read 2610.39278

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.39278 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.39278 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.39278 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co