Δτ-Bench:推動智能體適應不斷變化的能力
Δτ-Bench: Pushing Agents Toward a Moving Capability
Δτ-Bench 評估智能體適應持續演變的工具生態系統的能力,並提出 Steward 透過增量技能維護與動態配置工具,提升智能體在環境變化下的持續成功率。基準包含 42 個領域、1,588 項經驗證的多輪任務,涵蓋六個連續生態狀態;評測中表現最強的模型或 harness 任務成功率僅為 42.3%。實驗顯示,Steward 在降低技能維護開銷的同時,能在生態變化期間維持較高成功率。
Published on Oct 25
Authors:
,
,
,
,
,
,
,
Abstract
Δτ-Bench evaluates agent adaptability to evolving tool ecosystems, and Steward improves sustained success through incremental skill maintenance and dynamic provisioning.
In this paper, we introduce Δτ-Bench, an executable agent benchmark that models the structural evolution of large tool ecosystems. In Δτ-Bench, independently evolving services continually reshape how tasks can be accomplished: familiar tools may disappear or be replaced, while more efficient alternatives emerge amid overlapping functionality and noisy documentation. Against this moving capability frontier, we evaluate whether agents can proactively discover efficient tool-use paths and promptly revise their accumulated procedural knowledge, bringing together demands that existing benchmarks fail to examine. Δτ-Bench comprises 1,588 validated multi-turn tasks across 42 domains and six successive ecosystem states generated through capability-preserving transformations. Our evaluation shows that even the strongest evaluated model or harness achieves only 42.3% task success, while initial gains from existing skill-learning methods can not consistently translate into better adaptation as the ecosystem evolves. These findings motivate Steward, an ecosystem manager that jointly manages tools and procedural knowledge through incremental skill maintenance and dynamic provisioning during task execution. Experiments show that Steward sustains higher task success across ecosystem changes with lower skill-maintenance overhead, while improving both recovery from disrupted workflows and adoption of more efficient alternatives.
View arXiv page View PDF GitHub Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.34238 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.34238 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.34238 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co