跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 12 天前AI 評分45

BirdDreams:聯合生成透視影片與 BEV 影片的駕駛世界模型

BirdDreams: Joint Perspective and BEV Video Generation for a Plausible, Reactive Driving World Model

AI 導讀

BirdDreams 是一個自回歸駕駛世界模型,可聯合生成透視影片與鳥瞰視角(BEV)影片,將場景佈局納入預測,並以預測的 BEV 幾何資訊控制自車軌跡。在 PhysicalAI-AV 和 nuScenes 上,它比先前的駕駛世界模型更貼合輸入的自車軌跡;不需自車動作條件時,亦能生成更合理的自車移動。推理時,BirdDreams 只需單張相機影像,毋須 HD 地圖或車輛狀態輸入。

正文

Abstract

Driving world models aim to simulate how a scene unfolds under different ego actions. This requires plausible motion of surrounding vehicles as well as precise ego-trajectory control. Learning vehicle motion from perspective video alone is challenging because motion is entangled with appearance and vehicles can be occluded. Prior work therefore conditions video generation on layouts that specify vehicle motion, providing control over surrounding vehicles but requiring their future trajectories as inputs. We introduce BirdDreams, an autoregressive driving world model that jointly generates perspective video and a schematic bird's-eye-view (BEV) video, making scene layout part of the predicted future. The BEV, rendered from scene annotations for training, depicts roads and vehicles as simple shapes at a fixed metric scale, providing a top-down view of their positions and motion with reduced occlusion between vehicles. Perspective video, by contrast, retains scene details that the BEV omits, such as traffic signals and object appearance. Joint attention between two video diffusion transformers, one per stream, lets each stream inform the other during generation. The predicted BEV also provides a geometric interface for ego control: we translate and rotate its latest frame along an input ego trajectory and use the resulting frames as a soft condition that shows how the scene would appear to move if only the ego vehicle moved. At inference, BirdDreams generates both videos from a single camera image without requiring an HD map or vehicle-state inputs. On PhysicalAI-AV and nuScenes, BirdDreams follows input ego trajectories more faithfully than prior driving world models and generates more plausible ego motion without ego-action conditioning. Human evaluation shows that BirdDreams outperforms a perspective-only baseline in generating realistic surrounding-vehicle behavior.

View arXiv page View PDF Project page Add to collection

Get this paper in your agent:

hf papers read 2611.14815

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.14815 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.14815 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.14815 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co