跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 20 天前AI 評分58

Collaborative Intelligence Benchmark 衡量 AI 智能體的協作能力

Measuring AI Agents as Teammates with Collaborative Intelligence

AI 導讀

該研究提出 Collaborative Intelligence Benchmark,以 240 項任務評估 AI 智能體的協作能力。基準涵蓋 8 個團隊合作維度,評估 18 個封閉權重及開放權重模型,最高得分為 82.2/100。研究亦發現,能力分數相近的模型,作為隊友時的表現可相差 11 分;模型接手較弱模型作弊的任務時,通過率由 97% 降至 81%。

正文

Published on Nov 2

Authors:

,

,

,

,

,

,

,

Abstract

The Collaborative Intelligence Benchmark evaluates AI agents as teammates across 240 tasks, revealing that even top models have significant collaboration weaknesses unrelated to individual capability.

Progress in AI is often measured by what a single model can do on its own. Yet teams of AI agents are starting to make breakthroughs and discoveries in science, medicine, mathematics and software engineering. As AI begins to work in teams of agents and people, the ability to collaborate becomes a core part of their intelligence. Research on human teams has measured this ability for decades. Inspired by this, we introduce the Collaborative Intelligence Benchmark, which measures how well an AI agent works as a teammate. It contains 240 tasks across 8 dimensions of teamwork, from refusing an unsafe shortcut to checking the work that a weaker model leaves behind. Each task comes from a real record or an existing benchmark, from aviation accident reports and clinical cases to coding tasks and negotiations, and has a programmatically verifiable outcome. To isolate teamwork from individual capability, models are scored only on tasks that they can solve by themselves when they have full information. We evaluate 18 closed- and open-weight models and find that every model, across sizes and families, has clear weaknesses as a teammate, with the best model scoring only 82.2 out of 100. Frontier models almost always refuse unsafe shortcuts and report misconduct, yet they cave under interpersonal pressure, reaching the best overall deal in only 14% of conflicts, and their pass rate drops from 97% to 81% when they take over a task in which a weaker model has cheated. Crucially, general capability does not guarantee good teamwork, as models with nearly the same capability score differ by up to 11 points as teammates. The benchmark includes 160 public tasks with code and data, along with 80 held-out tasks for evaluation. As AI agents increasingly work alongside other agents and people, the ability to collaborate effectively is becoming a crucial capability to measure and develop.

View arXiv page View PDF Project page GitHub 15 Add to collection

Get this paper in your agent:

hf papers read 2611.02633

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.02633 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.02633 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.02633 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co