跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 13 天前AI 評分42

WM-PPO 如何透過評估後續狀態,結合世界模型學習行動

Learning to Act with World Models by Valuing What Comes Next

AI 導讀

WM-PPO 結合世界模型與 PPO 的價值評估器,利用預測的後續狀態改善任務成功率,無需特權資訊。在相若的真實環境互動步數下,ALFWorld 成功率由 PPO 的 63.5% 升至 87.5%,ScienceWorld 則由 39.3% 升至 52.9%。推理時只使用訓練所得的策略。

正文

Published on Nov 9

Authors:

,

,

,

Abstract

WM-PPO combines a world model with PPO’s critic to create auxiliary policy advantages and value targets from predicted successor states, improving task success without privileged information.

A sparse terminal reward tells an agent little about its steps. Much of the missing information is implicit in the environment's dynamics, because each state transition can change the conditions for success long before the reward arrives. A world model trained on real transitions can learn to predict what each action changes, but a predicted observation does not by itself show whether success has become more likely. With a binary terminal reward, PPO's critic makes this judgment by estimating the current policy's probability of success, but only for states the agent reaches. Our method, WM-PPO, elegantly brings the two together. For each action the policy samples, executed or not, the world model predicts the next observation and the critic values the predicted successor. In the policy channel, comparisons among these values give the policy auxiliary advantages; in the value channel, their mean gives the critic an auxiliary target. A shared gate weights both channels according to how reliably comparisons with similar margins predicted real outcomes earlier in training. The policy, critic and world model all learn from the agent's own interaction, with no teacher or privileged information, and only the policy is used at inference. With a comparable number of real environment steps, WM-PPO raises success from PPO's 63.5% to 87.5% on ALFWorld and from 39.3% to 52.9% on ScienceWorld. It matches the strongest PPO refinements we tested on ALFWorld and exceeds the best of them by 6.5 points on ScienceWorld. The trained policy more often completes prerequisite steps before placing an object or committing to an answer.

View arXiv page View PDF GitHub 1 Add to collection

Get this paper in your agent:

hf papers read 2611.12200

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 1

nuskonghou/wm-ppo

Reinforcement Learning •

Updated about 17 hours ago

•

1

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.12200 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.12200 in a Space README.md to link it from this page.

Collections including this paper 1

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co