用於檢索增強大語言模型搜尋狀態感知的自我評估強化學習
Self-Assessment Reinforcement Learning for Search-State Awareness in Retrieval-Augmented Large Language Models
Self-Assessment Reinforcement Learning(SARL)讓模型在訓練期間識別搜尋狀態、改善搜尋行為,無須在推理時額外評估。雙重獎勵設計鼓勵有效而高效的搜尋軌跡,SARL 較最強基線相對提升 24.3%,在多跳推理及深度研究基準上平均分別提升 35.9% 和 27.9%。作為獨立強化學習框架,SARL 在延長搜尋過程中仍能保持訓練穩定。
Published on Oct 21
Authors:
,
,
,
,
,
,
,
Abstract
Self-Assessment Reinforcement Learning trains search agents to recognize missing information and improve search efficiency without extra inference-time assessment.
Recent advances in search agents have enabled large language models to tackle complex tasks by retrieving and synthesizing external information. However, existing search agents often lack search-state awareness, i.e., the ability to recognize what information has been obtained and what remains missing, leading to hallucinated reasoning and prematurely terminated search. To address this limitation, we propose Self-Assessment Reinforcement Learning (SARL), a reinforcement learning framework in which models learn to recognize their own search state during training and improve their search behavior accordingly, without requiring additional assessment at test time. SARL achieves this through a dual reward design that captures the corrective effect of each assessment while penalizing excessive reliance on it, jointly encouraging effective and efficient search trajectories. Under this setup, SARL achieves a 24.3% relative improvement over the strongest baseline, with average gains of 35.9% on multi-hop reasoning and 27.9% on deep research benchmarks against the strongest per-benchmark baselines. SARL further surpasses methods built on auxiliary supervision or multi-agent coordination, despite operating as a standalone RL framework with no inference-time scaffolding. Beyond final performance, SARL maintains training stability under extended search horizons, a regime where existing methods consistently degrade, enabling sustained gains as task complexity grows.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2610.30435
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 1
Text Generation • 8B • Updated 1 day ago • 102Taewhoo/self-assessment-qwen2.5-7b-step325
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.30435 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.30435 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co