跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 11 天前AI 評分44

SO-Arena:評測 System One 模型的能力範圍與可靠性

SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models

AI 導讀

SO-Arena 評測 System One 模型在不同領域的能力範圍與決策可靠性,涵蓋 11 個領域及五類能力。對 21 個模型的評測顯示,擴展規模和增加推理帶來的效益因領域而異;整體表現最佳的模型在可靠性排名中僅列第七。

正文

Published on Nov 11

·

Submitted by

Qi Li

on Nov 13

Authors:

,

,

,

Abstract

SO-Arena evaluates System One models across domains and capability families, revealing that scaling and reasoning gains vary and that high overall accuracy does not ensure reliable decision-making.

System One models promise fast, general-purpose decision-making without lengthy token generation. However, understanding their capability spectrum and decision reliability requires more than aggregate performance on current benchmarks. Existing evaluations often provide limited insight into the breadth of model capabilities and the reliability of their decisions. We introduce SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families. For the capability spectrum, SO-Arena offers evidence-grounded evaluation through reliable sources and manual verification of reference labels against model-visible inputs; fine-grained capability assessment through domain-specific difficulty criteria; and realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks further enable system one-two comparison. Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option is available. Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains. More importantly, strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. Together, these findings highlight the need to characterize capability spectrum and decision reliability as complementary dimensions, providing a systematic view of the strengths and limitations of System One models.

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2611.15777

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.15777 in a model README.md to link it from this page.

Datasets citing this paper 1

SO-Arena/SO-Arena

Viewer •

Updated 27 minutes ago

•

1

•

41

•

2

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.15777 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co