題目會飽和，評審不會：模型評分下的排行榜代表甚麼

This is part three of a short series on reading AI evaluations. Part one, Same Model, Same Scaffold, 10 Runs, measured repeats: one configuration, ten runs, a 2.2–6.0 point window. Part two, 62% vs 58% on the Same Benchmark, measured task counts. Both assumed a stable instrument. This part asks what happens when the instrument is a model.

A 0–5 leaderboard puts model A at 4.1 and B at 3.7. Four tenths of a point — enough for a tweet that A leads, enough for B's team to rewrite prompts. Now learn that the minimum detectable difference (MDD) for such leaderboards at native item counts is 0.41 to 1.24 points, and that on the paper's general rubric, 0.4 does not clear it. What do you say then?

Atul Anand's Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM (arXiv:2609.27787, v1, preprint, August 17, 2026) decomposes 373,019 judge calls across 4 item sets, a 9-judge panel, and 42 systems. Its conclusion: when a model scores, neither item count nor run count is the variable that matters most.

The key is which question, not how good

A pointwise question asks what score an answer deserves; a pairwise question asks which of two answers is better. Most leaderboards use the first because it ranks cleanly, but they are not equally hard: "which is better" is far better conditioned, while "what score" folds the judge's scale habits, severity, and wording into the error.

Generalizability theory splits score variance into system, item, judge, and interactions such as system × judge, then computes Eρ², the share that is signal: rerun and the ranking holds, from 0 (noise) to 1 (repeatable). It is not accuracy; a judge can be stable and consistently wrong.

Why the ceiling does not rise with item count

This is the counterintuitive part:

"under a single judge, generalizability asymptotes to σ²s/(σ²s+σ²sj) regardless of item count, because the system-by-judge term carries no n_i. Items saturate; judges do not."

With one judge, the ceiling is σ²s/(σ²s+σ²sj) however many items you add: system × judge enters the error as σ²sj/nj, with judges in the denominator. Items shrink only the item-sampling term. What remains is "this judge is generous to A and harsh to B," which never cancels: one judge scores every item.

Two numbers pin it down. On MT-Bench at 0.75, 2 judges × 30 items roughly equals 1 judge × 77 items. At 0.80 nothing suffices: 0.80 sits 0.006 below the single-judge ceiling, and approaching it needs a divergent item count.

The advice splits in three. More items do not help within a fixed pool. More judges only approach the ceiling. Raising it needs a different judge universe, a different protocol, or item selection by discriminative power — the only item-side lever that raises it, to a 16.6% system variance share against Arena-Hard's 19.2%. Reading "items don't help" as "item design doesn't matter" inverts the point.

What 0.986 is, and what 8.6 points are

0.986 is often misread as 98.6% accurate. It is the reliability ceiling one judge reaches under a pairwise protocol that judges each pair in both orders and treats position as a factor. σ²sj falls two orders of magnitude below σ²s and the ceiling rises from 0.600–0.798 to 0.986 (bootstrap [0.934, 1.000], 11 systems); one judge reaches Eρ² = 0.93 at 24 items, so "one judge suffices."

But 0.986 means "rerun and the ranking holds," not "the judgment is right." On SummEval, reliability rose 0.711 → 0.959 (+0.248) while Kendall τ against a three-expert consensus moved only 0.596 → 0.710 (+0.114). Reliability is necessary for validity, not sufficient.

Then the 8.6 points, a new problem pairwise buys:

"A system placed in the first slot wins 8.6 percentage points more often than the same system, judged by the same judges on the same items, placed second."

First slot is presentation order, not first place: same system, same judges, same items, yet going first alone lifts the win rate 8.6 points. The bias varies 7.3 points across systems, reaches 25 at worst, and does not average away. Across 53 published win-rate comparisons the median claimed gain is 7.0 points; position bias is 1.23 times that, so an uncontrolled pairwise evaluation imports a bias larger than the median effect it hunts. In the same data, 70% of win-rate claims fall below pairwise's floor at native item counts.

An audit of 628 papers

The audit covers 628 arXiv papers from 2024–2026: five preregistered queries, 320 sampled by fixed seed, 312 coded, 227 reporting judge comparisons, κ = 0.73. Fewer than a quarter say whether they ran more than once (16.7% human, 20.0% automated); 46–67% report uncertainty; and all 17 checkable MT-Bench gains fall below its own floor (0.63 at 30 items, 0.54 at the native 80), which barely drops with more items because items saturate. This does not allege fabrication: the differences sit below the instrument's resolution, and the papers omit what a reader needs to notice.

What to do when you run a judge yourself

1. Report the judge universe: models, versions, count; otherwise reliability claims float.
2. Use the benchmark's native protocol and name it. Under MT-Bench's own template, 1–10 scale, and reference answers, the system variance share rises from 4.8% to 8.4% and the judges needed for 0.8 fall from 18 to 3.
3. Spend on judges before items. For 0.8 under a general 0–5 rubric: MT-Bench 25 judges, AlpacaEval 2 seven, the selected set and Arena-Hard two each.
4. Report MDD beside delta: the floor is 0.41–1.24 on a 0–5 scale, while 60 recovered pointwise gains have a median of 0.28; below it is not "leading."
5. When building a benchmark, select items by discriminative power.
6. Disclose repeat runs and response caching. Caching makes repeats fake.

With pairwise, run both orders and report prompt-level agreement: one judge rerun unchanged at temperature 0 moved the total about 1 point while 21.2% of items flipped win/tie/loss (arXiv:2610.30512).

What this paper cannot show

Six things, at least.

First, it is a single-author preprint v1, not peer reviewed.

Second, data and code are available from the author, not a public artifact: tables check out, reproduction is not cheap.

Third, it covers only 4 item sets, one pointwise 0–5 rubric, one scale; the paper concedes the general rubric is the conservative case.

Fourth, the Arena-Hard cell holds 17 items (others 32–43), making its σ²s estimate least precise; the pairwise arm uses the authors' own items, judge, and fixed baseline, not Arena-Hard's or AlpacaEval 2's pipeline.

Fifth, the audit samples rather than censuses: 628 framed, 312 coded, 227 reporting comparisons, κ between 0.45 and 1.00. It documents missing disclosure, not incorrect conclusions.

Sixth, reliability is not validity. A stable judge can stably measure the wrong thing, as SummEval shows.

The ruler

Back to 0.4 points: by this ruler it sits below the noise floor of most judge-scored leaderboards, and "who is stronger" is not answerable yet. Keep, not "0.4 points doesn't count," but "Items saturate; judges do not."

Adding items writes more marks on the ruler; changing judges makes the marks finer; changing the protocol swaps the ruler. Most people only do the first.

Disclosure: drafted with AI assistance and edited by the author.