跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-07AI 評分48

Coding-Agent Benchmarks Should Match Their Users' Task Flows

Coding-Agent Benchmarks Should Match Their Users' Task Flows

AI 導讀

研究團隊收集 JetBrains IDE 內軟件工程師的 4,782 段 Agent 工作階段,指出 Coding-Agent 基準評測應按目標用途配合用戶的任務流程。

正文

Published on Oct 7

Authors:

,

Abstract

The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In a pilot on 700 SWE-Bench Pro tasks, solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation.

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.09633

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.09633 in a model README.md to link it from this page.

Datasets citing this paper 2

JetBrains-Research/SWE-TaskFlow-SWE-Bench-Pro

Viewer •

Updated about 22 hours ago

•

4.08k

•

1.15k

JetBrains-Research/SWE-TaskFlow-SWE-Bench-Pro-Trajectories

Viewer •

Updated about 22 hours ago

•

25.2k

•

44

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.09633 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co