跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-17AI 評分43

決策模型控制器增加智能體成本,卻未提升成功率

Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success

AI 導讀

Jev 1.13 決策模型控制器用於 Claude Sonnet 5.5 終端智能體時,未提升成功率,反而增加每次嘗試成本。在 Terminal-Bench 2.1 的 21 項留出任務、315 次試驗中,Jev 的成本高 19%,通過率則與對照組同為 73.0%。Jev 判斷任務是否完成的平均概率為 0.50,但這些運行中有 87% 通過,顯示其校準不佳,會要求智能體核實已正確完成的工作。

正文

Published on Oct 17

Authors:

Abstract

A typed decision model consulted at fixed points increased agent costs without improving success rates, despite better ranking of finish requests, due to poor calibration.

Agent scaffolds increasingly delegate small decisions (whether to stop, verify, or change approach) to cheaper models. We ask whether a typed decision model, Jev 1.13, consulted only at deterministic decision points, makes a Claude Sonnet 5.5 terminal agent cheaper without making it worse. In a pre-specified study on Terminal-Bench 2.1, we hold one agent loop fixed and vary only the controller: none, deterministic rules, a small LLM (Claude Haiku 5.5) answering the same questions, Jev, and a placebo that samples Jev's development-phase choice frequencies at the same moments. On 21 held-out tasks (315 trials), Jev raised cost per attempt by 19% (geometric-mean ratio 1.19, 95% CI 1.02-1.36) at an identical pass rate (73.0% vs. 73.0%). Jev was cheaper than the small-LLM controller (ratio 0.81) but 22% more expensive than the placebo, and no pass-rate difference between arms was detectable. Offline, Jev ranked finish requests better than the alternatives (AUROC 0.78 vs. 0.62 for rules and 0.60 for the small LLM) but was poorly calibrated: its mean probability that a run was complete was 0.50 when 87% of those runs passed, so it asked the agent to verify work that was already correct. No controller ever chose to abandon a run. Informative decision signals did not become useful interventions; we discuss what that would require.

View arXiv page View PDF GitHub 0 Add to collection

Get this paper in your agent:

hf papers read 2610.24649

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.24649 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.24649 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.24649 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co