跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 24 天前AI 評分48

HALO:以人類為基礎的主動式智能體細粒度評估

HALO: Human-Grounded Granular Evaluation of Proactive Agents

AI 導讀

HALO 推出以人類標註為基礎的主動式智能體評測基準及數據集,結果顯示過度主動是主要失誤模式。HALO-Arena 涵蓋 4 類需求、10 個日常領域及 50 個個人環境;HALO-8K 收錄 10 個模型在 400 個情境中的 8,000 個回應,由 20 位專家標註。最佳閉源及開源模型的可接受度分別為 72.88% 和 56.63%,被拒回應中有 61.20% 涉及時機過早或介入力度過大。

正文

Published on Oct 29

Authors:

,

,

,

,

,

,

Abstract

HALO introduces a human-annotated benchmark and dataset to evaluate proactive agents, revealing that over-proactivity is the dominant failure mode.

Proactive agents must decide whether, when, and how far to assist without an explicit request. Existing evaluations often rely on automated scores or binary verdicts, offering limited human diagnosis of evidence use and intervention quality. To address these gaps, we introduce HALO, Human-Grounded GrAnular EvaLuation of PrOactive Agents. HALO-Arena covers 4 proactive need types, Dependency, Blockage, Risk, and Opportunity, across 10 everyday domains and 50 personal environments. Scenarios require agents to retrieve evidence from structured databases or unstructured documents before judging whether assistance is warranted and choosing its scope. On HALO-Arena, we construct HALO-8K, a dataset of 8,000 responses from 10 models across 400 scenarios, annotated by 20 experts for action level, acceptability verdicts, and human diagnosis. For rejected responses, human diagnosis covers timing, intensity, and other issues. Recorded reasoning tokens and retrieval rounds measure the effort behind these responses. The best proprietary and open-source models achieve 72.88% and 56.63% acceptability, respectively. Our central finding is that over-proactivity is the dominant observed failure mode, with Premature timing or Excessive intensity accounting for 61.20% of rejected responses. Further analyses yield complementary findings on the balance between intervention and restraint, evidence acquisition, and the calibration of timing and intensity.

View arXiv page View PDF GitHub Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.39086 in a model README.md to link it from this page.

Datasets citing this paper 1

CedPei/HALO-8K

Viewer •

Updated 3 days ago

•

8k

•

173

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.39086 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co