跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-09AI 評分48

AgentSWE:編碼智能體能否建出你真正需要的智能體?

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

AI 導讀

AgentSWE 基準評估編碼智能體能否按需求建立智能體;結果顯示,受評智能體無一能接近要求。基準設有 25 項任務,涵蓋按自然語言需求建立完整智能體、編輯正式環境中的智能體程式碼庫,以及按可量度目標優化。常見失敗包括過度配合可見案例、程式碼看似完整但執行環境空洞,以及無根據地宣稱完成。

正文

Published on Oct 9

Authors:

,

,

,

Abstract

As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their users' own workflows. This delegation carries a quiet assumption, that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. No existing benchmark tests that assumption. We introduce AgentSWE, a benchmark for agent software engineering spanning three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target. Each of its 25 tasks is framed as a commission rather than an exam: the builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement, grading behavior against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective. Evaluating a range of coding agents across models and harnesses, we find that none comes close to delivering what a requirement asks for: the gap between what a builder demonstrates and what it delivers is the norm, not the corner case. An accompanying empirical analysis of the failure trajectories identifies recurring failure modes, among them overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion, each a signal by which coding agents can learn to build the agents they are asked for.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.13361

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.13361 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.13361 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.13361 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co