跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 11 天前AI 評分56

Termination Lag:編碼智能體解題後仍持續執行

Termination Lag: Coding Agents Solve Early but Stop Late

AI 導讀

研究追蹤 5 個 LLM、2 種智能體框架及 3 個 SWE-bench 數據集,發現編碼智能體常在產生正確補丁後仍繼續執行。38–82% 的成功執行在首次原始碼修改時已得到正確補丁;核心設定下,成功執行中位數有 35–60% 的步驟是在正確補丁產生後執行。這些額外步驟主要用於驗證和探索,1–8% 的個案會把已正確的解答改壞;研究認為,改善停止判斷或可減少不必要計算。

正文

What if coding agents already solved the problem but don't know when to stop?

We studied 5 LLMs, 2 agent frameworks, and 3 SWE-bench datasets, tracking and evaluating intermediate code patches throughout agent execution.

🔍 Key findings:

  • 38–82% of runs that reach a correct patch get it right on their first source-code edit.
  • A median of 35–60% of steps in successful runs occur AFTER the correct patch is produced in our core settings.
  • Agents spend most of these extra steps verifying and exploring.
  • In 1–8% of cases, agents even break an already-correct solution and finish with an incorrect patch.

These findings suggest an important opportunity: making coding agents recognize when to stop could significantly reduce unnecessary computation.

💬 How should coding agents decide when a solution is good enough to submit?

Github:
https://github.com/musacim/termination-lag

HF:
https://huggingface.co/datasets/musacim/termination-lag-trajectories

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co