DeepStress:對深度搜尋智能體進行壓力測試
DeepStress: Stress-Testing Deep Search Agents
DeepStress 是一套動態壓力測試框架,透過可控地降低檢索證據品質,測試搜尋智能體在不同證據條件下的表現。研究以 HotpotQA 和 BrowseComp-Plus 的問題評估 12 個系統,發現多個開放權重智能體在來源有明顯可信度警號時仍會自信作答。許多 HotpotQA 問題即使所有檢索文件均有事實錯誤,仍會依靠模型參數中的知識作答。
Published on Nov 12
Authors:
,
Abstract
A dynamic stress-testing framework shows that search agents often answer confidently despite unreliable retrieved evidence and may rely on memorized knowledge when documents are factually wrong.
While search agents demonstrate impressive capabilities in multi-step question answering, their robustness to poor-quality evidence remains under-explored. Such evidence is rare in benchmark corpora, and resulting failures are hard to reproduce. We propose DeepStress, a dynamic stress-testing framework that subjects agents to degraded evidence at controlled rates. It replaces the retrieval module with a synthetic environment where documents are generated dynamically for each query, under independent control of their trustworthiness, relevance, and factuality. To characterize behavior under stress, we go beyond binary accuracy with a five-way classification of final-answer outcomes and two new reliability-aware metrics, RAS and CoPRAS, which credit cautious behavior and account for computational cost. Evaluating 12 systems on questions from HotpotQA and BrowseComp-Plus, we find that several open-weight agents answer confidently when sources contain obvious credibility red flags, and that many HotpotQA questions are answered from parametric knowledge even when all returned documents are factually wrong.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2607.13920
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2607.13920 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2607.13920 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2607.13920 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co