OneWM:統一因果訓練下,世界建模與動作建模能否互相促進?
Can World and Action Modeling Benefit Each Other under Unified Causal Training?
OneWM 以統一因果訓練結合世界建模與動作建模,交替預測動作和生成未來畫面,並利用自身預測延續場景推演。頁面展示真實機械人示範及由初始場景和腕部鏡頭觀察開始的想像推演;初始觀察後的畫面由模型生成,預測動作不會在模擬器中執行。
OneWM — official resources
Real-world robot demonstrations
Full robot recordings at original speed, without audio.
Block stacking — stacking three colored blocks.
Storage — inserting a blue block into a storage unit.
Breakfast preparation — plating toast and a fried egg.
An Endless Dream
An imagined rollout starting from initial scene and wrist-camera observations. OneWM alternates action prediction and future-frame generation, using its own predictions to continue the rollout. Frames after the initial observations are model-generated; the predicted actions are not executed in a simulator.
Alphabet soup into the basket
Two objects into the basket
The first 16 seconds at original speed, with synchronized scene and wrist views plus predicted actions: the relative control path, translation, rotation, and gripper values. The path accumulates saved translation commands within each round; it is not a measured robot trajectory.
More action-conditioned world predictions and interactive visualizations are available on the project page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co