跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 3 天前AI 評分53

TwinLab-Bench 評估 AI 科學家在真實實驗室智能體孿生環境中的表現

TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs

AI 導讀

TwinLab-Bench 是一項評估 AI 科學家自行設計實驗能力的基準,涵蓋物理科學、生命科學和工程領域的 52 項任務。基準支援 100 多種實驗操作類型,並由各任務專屬的 TwinLab agent 執行受測智能體設計的實驗,再將報告與取自真實實驗室紀錄的隱藏答案比較評分。研究顯示,前沿模型在這些任務上仍有困難,表現最佳的 Claude Fable 5.2 平均得分為 45.0。

正文

Published on Nov 19

Authors:

Abstract

AI agents are increasingly expected to act as AI scientists, which requires three increasingly open abilities: carrying out a given computational workflow correctly, choosing the next measurements from a given design space, and designing their own experiments. Existing benchmarks mainly evaluate the first two abilities, in a terminal or in a closed loop that selects from predefined candidates. Evaluating the third ability requires reliably executing an experiment that an agent designs, with readings grounded in a real laboratory. We introduce TwinLab-Bench, which evaluates AI scientists on self-designed experiments in 52 tasks from laboratories in the physical sciences, the life sciences, and engineering. The benchmark supports more than 100 experimental action types, and the agent's report is scored by comparison with a hidden answer taken from real laboratory records. Each task has its own TwinLab agent, an agent that serves as the environment and executes the experiments that the tested agent designs, reading the raw materials of the laboratory and calling calculators that are built from its records. Compared with a language model that sees only the task description, the TwinLab agent almost eliminates readings that deviate from the real records and inconsistent answers to a repeated request, and closed-book probes control for answers recalled from memory. We find that these tasks remain hard for frontier models: the strongest, Claude Fable 5.2, averages a score of only 45.0, and we conduct an empirical study of where it fails.

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2611.26248

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.26248 in a model README.md to link it from this page.

Datasets citing this paper 1

ZJUSCL/TwinLab-Bench

Viewer •

Updated about 3 hours ago

•

48

•

14

•

1

Spaces citing this paper 1

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co