跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 19 天前AI 評分41

DreamX:影片擴散模型是通用潛在動作學習器

DreamX: Video Diffusion Model is a Universal Latent Action Learner

AI 導讀

DreamX 利用預訓練影片擴散模型學習精簡且不受外觀影響的潛在動作,以支援跨具身遷移和高效策略學習。模型將每個影片畫格壓縮 300 倍,由 1,200 個 token 減至 4 個,並在 LARY-Bench 的潛在動作模型中達到最先進表現。該潛在空間支援不同具身與環境之間的任意影片遷移及零樣本動作遷移。

正文

Published on Nov 3

Authors:

,

,

,

,

,

,

,

Abstract

DreamX leverages a pretrained video diffusion model to learn compact, appearance-invariant latent actions that enable cross-embodiment transfer and efficient policy learning.

Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by 300times (1{,200 rightarrow 4} tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.

View arXiv page View PDF Project page Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.05377 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.05377 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.05377 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co