跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 15 天前AI 評分45

Longtail-SWE:為未充分覆蓋的程式語言打造高品質 SWE 任務基準

Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages

AI 導讀

Longtail-SWE 推出涵蓋 13 種程式語言的 SWE 基準,包含 202 個問題解決任務,補足現有程式碼庫層級基準對這些語言覆蓋不足的情況。研究以 solver、judge 和 repairer 流程檢視並修訂任務描述及測試,並評估九種模型配置的解題表現、重複嘗試的一致性及資源使用。基準、可執行環境、建構與評估工具、Agent 軌跡及較容易的任務集均已開放。

正文

Published on Nov 6

Authors:

,

Abstract

Challenging repository-level benchmarks remain limited for many programming language ecosystems, where filtering scarce candidates with ambiguous descriptions or defective tests can discard valuable engineering problems. We introduce Longtail-SWE, a benchmark of 202 issue-resolution tasks across 13 languages with limited coverage in existing repository-level benchmarks: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Its construction uses a solver--judge--repairer pipeline: a solver attempts each candidate task, and a judge reviews the task together with the solver's trajectory and test results to guide revisions to tests and task descriptions. These revisions aim to correct tests that reject valid alternative implementations or accept incorrect solutions while retaining challenging tasks and preserving their intended requirements. We evaluate nine model configurations, including frontier models, and analyze task-solving performance, consistency across repeated attempts, and resource usage. We release all benchmark instances, executable environments, complete construction and evaluation tooling, and agent trajectories, together with an additional set of easier tasks to support evaluation of smaller models.

View arXiv page View PDF Project page GitHub 0 Add to collection

Get this paper in your agent:

hf papers read 2611.10304

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.10304 in a model README.md to link it from this page.

Datasets citing this paper 2

nebius/Longtail-SWE

Viewer •

Updated about 4 hours ago

•

471

•

6

•

1

nebius/Longtail-SWE-trajectories

Viewer •

Updated about 4 hours ago

•

5.45k

•

3

•

1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.10304 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co