跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-21AI 評分43

ALPS:世界模型的視覺狀態編碼器

ALPS: Visual State Encoders for World Models

AI 導讀

ALPS 是預訓練視覺狀態編碼器,整合物理狀態預測、語言對齊與像素重建,作為世界模型的共用基礎。以靜態圖像和文字從頭預訓練的 ViT-B/16,在 15 個公開編碼器的統一評測中取得最低 LIBERO 狀態探測誤差,媲美規模大 3.5x 的 V-JEPA 2-L。

正文

Published on Oct 21

Authors:

,

,

,

,

,

,

Abstract

ALPS is a pretrained visual encoder that unifies physical-state prediction, language alignment, and pixel reconstruction to serve as a shared foundation for world models.

World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action in world models.

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.30120

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 1

UCSC-VLAA/ALPS

Image Feature Extraction •

Updated 32 minutes ago

•

1

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.30120 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.30120 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co