跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 27 天前AI 評分31

長時程 LLM 智能體的多尺度策略最佳化

Multiscale Policy Optimization for Long-Horizon LLM Agents

AI 導讀

多尺度策略最佳化(MPO)透過在回合尺度對齊信用分配與策略更新,改善長時程 LLM 智能體的訓練。MPO 結合多尺度優勢估計(MAE)與回合層級重要性抽樣,在使用 4B 和 30B-A3B 模型進行的搜尋、工具使用及自動化任務實驗中,持續勝過多個強化學習基線。

正文

Published on Oct 26

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Multiscale Policy Optimization improves long-horizon agent training by aligning credit assignment and policy updates at the turn scale.

Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We propose Multiscale Policy Optimization (MPO), combining Multiscale Advantage Estimation (MAE) and turn-level importance sampling to align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.

View arXiv page View PDF Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.35013 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.35013 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.35013 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co