編碼智能體持續演進能否避免災難性遺忘?研究發現後續任務會削弱舊任務表現
Can Coding Agents Sustain Continual Evolution without Forgetting?
一項關於編碼智能體持續 harness 自我演進的實證研究發現,後續任務演進會令先前數據集上的表現大幅下滑。研究將三個軟件工程任務數據集整理為六種演進方向,並以兩種代表性 harness 演進方法進行持續演進實驗。
Published on Nov 2
Authors:
,
Abstract
Empirical study shows continual harness self-evolution across software engineering tasks causes severe catastrophic forgetting, with performance on earlier datasets dropping sharply after later evolution.
Recently, harness self-evolution has emerged as an important paradigm for improving agent performance. Prior studies show that evolving the agent harness can effectively enhance agent capabilities without modifying the foundation model's parameters. We find that existing approaches largely study agent evolution on a single task. In practice, coding agents may need to handle multiple task types over time. Whether an agent harness can continuously evolve across multiple tasks without suffering from catastrophic forgetting remains an important open question. In this paper, we investigate this open question through a comprehensive empirical study of continual harness self-evolution. We organize three software engineering task datasets into six different evolution directions and apply two representative harness evolution methods to perform continual evolution along these directions. Our experimental results show that evolution on a subsequent dataset can substantially undermine the performance on previously seen datasets. For example, with DeepSeek-V4-Flash-0731, Meta-Harness improves its performance on SWE-Bench to 94.0% after evolution. However, after further evolution on FeatureBench, its performance on SWE-Bench drops to 32.0% by the fifth round, corresponding to a 66% relative decrease. Finally, based on our experimental findings, we identify several insights, providing directions for the continual self-evolution of agent harnesses. Code is available at https://github.com/THU-Agent/continual-harness-evolution.
View arXiv page View PDF GitHub 0 Add to collection
Get this paper in your agent:
hf papers read 2611.03219
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.03219 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.03219 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.03219 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co