SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning
SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning
SERO 透過訓練時使用可執行的 Oracle 延續,改善長時程智能體強化學習,且推理時無額外開銷。研究在 ALFWorld 和 WebShop 上使用 Qwen2.5-1.5B-Instruct 及 Qwen2.5-7B-Instruct 評估;在 ALFWorld 上,學習完整延續比只監督根節點 Oracle 動作更有效,政策可達性加權亦提升分佈外表現。
Published on Oct 14
Authors:
,
,
,
,
,
,
Abstract
SERO improves long-horizon agentic reinforcement learning by training on executable oracle continuations from real policy states, weighted by policy compatibility, without inference-time overhead.
Long-horizon agentic RL is constrained by the futures visited by the current policy. When rollouts fail to reach useful behaviors, improved credit assignment cannot provide supervision for states and actions that were never encountered. On-policy distillation (OPD) introduces teacher guidance on policy-induced states, and recent multi-turn variants improve how this guidance is scheduled, replayed, refined, or validated. However, they do not directly use the oracle-controlled future itself as the training trajectory. We introduce State-Rooted Executable Rollout Optimization (SERO), which augments on-policy agentic RL with executable oracle continuations. SERO restores a state actually reached by the current policy and lets a training-only oracle continue interacting with the real environment, making the downstream observations and oracle actions themselves available for learning. It then applies policy-reachability weighting, which weights each oracle step by the frozen behavior policy's compatibility with the preceding oracle actions. This allows deeper supervision to become effective as the policy becomes more compatible with the oracle branch. The oracle loss is optimized separately from the base RL estimator and no oracle is used at inference time. We evaluate SERO on two challenging interactive-agent benchmarks, ALFWorld and WebShop, using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. On ALFWorld, learning the full executable continuation is substantially more effective than supervising only the root oracle action, while policy-reachability weighting further improves out-of-distribution performance.
View arXiv page View PDF Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.19397 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.19397 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.19397 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co