跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 12 天前AI 評分45

MMDM:大規模多任務決策能力測量

MMDM: Measuring Massive Multitask Decision Making

AI 導讀

MMDM 是一項涵蓋 16 類任務的基準測試,包含 5,448 條問題,選項數由 2 至 1,024,輸入最長達 64,817 tokens。研究評估 28 個決策模型,發現不少模型在選項邏輯、受約束決策及多選任務上表現較弱。

正文

Published on Nov 10

Authors:

,

,

,

,

Abstract

System-1 Decision Models directly make structured probabilistic decisions over query-specified options given a state in a single forward pass, without autoregressively generating reasoning trajectories or textual responses. Recent evaluations cover diverse tasks but often rely on existing benchmarks, where prior exposure to test material can obscure generalization. Moreover, real-world decision making often involves long contexts, large option spaces, and complex logical dependencies, challenges that existing benchmarks do not adequately capture together. Thus, we introduce MMDM (Massive Multitask Decision Making), a benchmark of 5,448 high-quality questions across 16 task categories, combining newly constructed tasks with source-grounded recompositions and rewrites for single-select, multi-select, and soft-label decision making in fresh contexts. It covers realistic scenarios such as agent state tracking, tool and skill selection, preference judgments, and rubric scoring, with option spaces ranging from 2 to 1,024 and inputs extending to 64,817 tokens. With MMDM, we conduct a comprehensive quantitative evaluation of 28 decision models, showing that many perform well on common knowledge yet struggle with interdependent options, constrained decisions, and multi-select tasks. For example, Perplexity Decider-V1.2 reaches 78.75% Hard Single-select Accuracy on world knowledge, but only 53.95% on option logic and 47.95% on constraint planning, while the best Hard Multi-select Accuracy across all models is 36.81%. These results highlight joint reasoning, constraint-aware selection, and complete multi-select recovery as key priorities for decision model development, with MMDM providing a testbed for measuring this progress.

View arXiv page View PDF Project page GitHub 1 Add to collection

Get this paper in your agent:

hf papers read 2611.14309

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.14309 in a model README.md to link it from this page.

Datasets citing this paper 1

shenjunhao/mmdm

Viewer •

Updated 22 minutes ago

•

5.45k

•

211

•

1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.14309 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co