跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 17 天前AI 評分50

TB-MathModel:首個面向智能體、可驗證的終端數學建模基準

Terminal-style Mathematical Modeling Bench: The First Agentic and Verifiable Benchmark for Mathematical Modeling

AI 導讀

TB-MathModel 提出一套面向 LLM 智能體的數學建模評測基準,讓智能體在容器化終端中處理 18 道 CUMCM/MCM 賽題。評測以最終問題的數值核心評分,使用 94 篇獲 Outstanding 獎的論文作參考,並以六類可判定不變式及 624 項 Lean 4 證明義務核查可行性。

正文

GPT-6 Astra and Claude Fable 5.1 barely pass!?

Today's LLM agents are strong when the problem is already well posed — write code, solve a math problem, call an API. The real world never hands you a problem that way. Nobody defines your variables, tells you which model to build, where the constraints are, how dirty the data is, or whether the conclusion survives contact with reality.

Compressing a messy real-world situation into a model that is computable, verifiable, and defensible — that is mathematical modeling. It is the interface layer an LLM agent must cross to go from solving exercises to doing real work, and it is the least adequately measured layer we have.

TB-MathModel is evaluation infrastructure built for exactly that layer.

We use CUMCM/MCM contest problems as the entry point, and not because we care about contests: they are one of the very few public corpora that bundle a complete, unstructured real-world problem with genuine attachments, official figures, and human expert solutions. Eighteen of them go straight into containerized terminals — read the statement, open the spreadsheets, look at the figures, write code, run the solver, emit one structured JSON. That is a miniature of real modeling work.

Three design choices make this layer measurable:

  1. We do not grade the report — we check the numbers. Each task anchors on the numeric core of the final question, scored direction-aware and one-sided: beating the human paper never costs you points.

  2. Not one reference answer — 94 of them. Modeling has no unique solution, so every Outstanding-award paper gets a vote: a model has to land close to many human experts at once, not memorize one answer.

  3. Feasibility verified independently, and machine-checkable. A programmatic gate over six decidable invariant forms plus 624 kernel-checked Lean 4 obligations — no Mathlib, no sorry. Reporting a number is not enough; it has to survive the physics.

The result: the best entry, GPT-6 Astra, reaches only 64.6%; Fable 5.1 reaches 63.1%; a $2.54 open model holds 59.8% at 1/100 the cost.

We are still far from agents that can handle real-world problems. And what that road needs is a uniform environment, a verifiable reward, and an agent-trajectory corpus that compounds — precisely what TB-MathModel sets out to provide.

Code and data: https://github.com/wxj630/terminal-bench-math-modeling

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co