Coding-Agent Benchmarks Should Match Their Users' Task Flows
Coding-Agent Benchmarks Should Match Their Users' Task Flows
研究團隊收集 JetBrains IDE 內軟件工程師的 4,782 段 Agent 工作階段,指出 Coding-Agent 基準評測應按目標用途配合用戶的任務流程。
Published on Oct 7
Authors:
,
Abstract
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In a pilot on 700 SWE-Bench Pro tasks, solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation.
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.09633
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.09633 in a model README.md to link it from this page.
Datasets citing this paper 2
JetBrains-Research/SWE-TaskFlow-SWE-Bench-Pro
Viewer • Updated about 22 hours ago • 4.08k • 1.15k
JetBrains-Research/SWE-TaskFlow-SWE-Bench-Pro-Trajectories
Viewer • Updated about 22 hours ago • 25.2k • 44
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.09633 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co