TwinLab-Bench 評估 AI 科學家在真實實驗室智能體孿生環境中的表現
TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs
TwinLab-Bench 是一項評估 AI 科學家自行設計實驗能力的基準,涵蓋物理科學、生命科學和工程領域的 52 項任務。基準支援 100 多種實驗操作類型,並由各任務專屬的 TwinLab agent 執行受測智能體設計的實驗,再將報告與取自真實實驗室紀錄的隱藏答案比較評分。研究顯示,前沿模型在這些任務上仍有困難,表現最佳的 Claude Fable 5.2 平均得分為 45.0。
Published on Nov 19
Authors:
Abstract
AI agents are increasingly expected to act as AI scientists, which requires three increasingly open abilities: carrying out a given computational workflow correctly, choosing the next measurements from a given design space, and designing their own experiments. Existing benchmarks mainly evaluate the first two abilities, in a terminal or in a closed loop that selects from predefined candidates. Evaluating the third ability requires reliably executing an experiment that an agent designs, with readings grounded in a real laboratory. We introduce TwinLab-Bench, which evaluates AI scientists on self-designed experiments in 52 tasks from laboratories in the physical sciences, the life sciences, and engineering. The benchmark supports more than 100 experimental action types, and the agent's report is scored by comparison with a hidden answer taken from real laboratory records. Each task has its own TwinLab agent, an agent that serves as the environment and executes the experiments that the tested agent designs, reading the raw materials of the laboratory and calling calculators that are built from its records. Compared with a language model that sees only the task description, the TwinLab agent almost eliminates readings that deviate from the real records and inconsistent answers to a repeated request, and closed-book probes control for answers recalled from memory. We find that these tasks remain hard for frontier models: the strongest, Claude Fable 5.2, averages a score of only 45.0, and we conduct an empirical study of where it fails.
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2611.26248
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.26248 in a model README.md to link it from this page.
Datasets citing this paper 1
ZJUSCL/TwinLab-Bench
Viewer • Updated about 3 hours ago • 48 • 14 • 1
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co