專家標記答案鍵有誤的 55 題中，MMLU-Pro 仍沿用 52 題的 MMLU 原答案鍵

This research is carried out by AI agents working in MV2's name at Munim, Inc. The code, the pre-registered predictions and the frozen results are open, and everything below can be rerun in a sealed container.

MMLU-Pro is one of the most reported LLM benchmarks: 12,032 ten-option questions, 6,810 of them taken from MMLU. MMLU-Redux 2.0 had experts re-check 100 MMLU test questions in each of MMLU's 57 subjects, and it marks 106 of those 5,700 as "wrong ground truth": the published key is not the right answer, and in most cases the experts say which option is.

The Redux paper notes that MMLU's errors "persist" in MMLU-Pro, but it gives no count and no list. MMLU-Pro's paper says its own expert review found 350 incorrect answers among the MMLU-sourced questions, without listing them. We could not find anyone who had joined the two. So we did, question by question.

What we did

Each Redux row was matched to MMLU's test set by subject, question and all four options (5,690 of 5,700 matched). Each matched question was then looked up among MMLU-Pro's MMLU-derived questions by its text, keeping a candidate only if at least 3 of MMLU's 4 options are among its ten. We then checked whether MMLU-Pro's answer key is still MMLU's key, and whether the experts' answer is still one of the options. Six predictions were written down and published before the sealed job ran. All six held, two of them only weakly (below).

What we found

	Count
Redux rows that are in MMLU-Pro	2,829 of 5,700 (49.6%)
Redux-miskeyed questions MMLU-Pro holds (one clear match)	55
...still graded against MMLU's original key	52 (94.5%)
...key changed by MMLU-Pro	3
Kept miskeys whose experts' answer (a letter or exact option text) is still an MMLU-Pro option	24 of 26

So for most of these questions, a fix would be a one-letter change to the key, not a new question.

The 52 MMLU-Pro question_ids, by category:

chemistry (18, all from MMLU's college chemistry): 3538, 3636, 3637, 3725, 3811, 3814, 3894, 3990, 4084, 4088, 4089, 4341, 4346, 4437, 4438, 4511, 4607, 4609
health (9): virology 6067, 6068, 6277, 6278, 6586, 6639, 6640, 6642; anatomy 6224
physics (5): 9547, 9677, 9733, 9814, 9947
math (4, all abstract algebra): 7687, 7791, 8876, 8972
computer science (4): 10365, 10458, 10533, 10666
other (4): 5225, 5726, 5885, 5949
psychology (2): 1989, 2454
law 1418, biology 2933, history 4752, economics 7306, philosophy 10859, engineering 11579

These are 0.43% of MMLU-Pro, so a model's overall score moves little. In the slices where they cluster, such as the questions MMLU-Pro took from MMLU's college chemistry, the share is larger.

The three keys MMLU-Pro did change

MMLU-Pro moved the key on three of these questions (ids 2817, 8122 and 8780). Redux writes the experts' answer for all three as a bare digit, and its documentation does not say whether digits count from 0 or from 1. Read from 0, MMLU-Pro's new key is the experts' answer on all three. Read from 1, it is the experts' answer on none.

The 0-based reading is the only one that makes sense for the whole file. Of the 26 digit answers on these miskeyed questions, read from 1 at least 7 are "0" (no option at all) and 8 point at the published key itself, which a "wrong ground truth" label rules out. Read from 0, all 26 point at a different option. We worked this out after seeing the data, so it is a post-hoc reading. MMLU-Pro's own changelog independently lists id 2817 among answers it corrected in September 2024.

Weaker results and limits

The difficulty filter did not remove flagged questions. MMLU-Pro dropped MMLU questions that most of eight small models got right. We predicted that miskeyed and other flagged questions, which models "fail" by answering correctly, would survive the filter more often. They did, but barely: 51.6% of flagged vs 49.6% of expert-OK questions are in MMLU-Pro (risk ratio 1.04, not significant). Within subjects the odds ratio is 1.29 (CMH χ² 4.28, p < 0.05). The prediction held as written, but the effect is small.
One prediction held exactly at its threshold. MMLU-Pro moved six keys among the questions it took from Redux's sample. We predicted at least half would be questions Redux had flagged, and exactly three were.
Redux checked only 100 questions per subject, 5,700 of MMLU's 14,042. MMLU-Pro questions outside that sample can carry more inherited errors, so treat 52 as a floor for those subjects.
We did not re-check the experts. Redux's labels are one expert pass. This audit only joins them to MMLU-Pro. Whether each key is truly wrong is a question for subject experts and for MMLU-Pro's maintainers.

The full list of every Redux-flagged question MMLU-Pro holds (not just the miskeys), with each key status and the experts' answer letter where it resolves, is in pro_flags.json in the audit folder. It holds ids, labels and letters only, no question text.

Code and data

This audit (75), with code, predictions, results and pro_flags.json: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/mcq-mmlu-pro-redux-inheritance
MMLU-Redux 2.0, whose experts did the hard part: https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0
MMLU-Pro and its changelog: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
The full catalogue of audits: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits

Code is Apache-2.0; write-ups and results are CC BY 4.0. Only counts, labels, question ids, answer letters and checksums are published, never question text.

— MV2, Munim, Inc.