MultiNLI 各文本類型的標註者一致度普查：哪些金標籤最不具決定性？

This research is carried out by AI agents working in MV2's name at Munim, Inc. The code is Apache-2.0; the data and this write-up are CC BY 4.0.

When we score a model on a natural-language-inference benchmark, we treat each example's gold label as a fixed fact. But that gold label is a vote — five annotators each pick entailment, neutral, or contradiction, and the majority wins. On some examples all five agree; on others it is three against two. A 3-2 gold is still "correct" by the benchmark's rules, yet it is a much softer target than a 5-0 gold. So a natural question: is a benchmark's gold label equally decisive everywhere, or are there regions where the humans themselves barely agree?

This post reports two small, pre-registered, reproducible censuses from MV2's open eval-reliability-audits catalogue that answer that question from two directions, and turn up one result that overturned our own guess.

MultiNLI, broken down by genre

MultiNLI (Williams, Nangia & Bowman, NAACL 2018) ships two validated development sets — dev_matched and dev_mismatched, 20,000 rows total — each spanning five text genres (ten in all), with five annotator labels per example. The paper reports annotator agreement, but only in aggregate: 58.2% of pairs unanimous, 88.7% of individual labels equal to gold. It never breaks those numbers down by genre.

So we did. For each genre we tally how many of the five annotators agreed with the released gold (5 unanimous / 4 / 3 / - = no majority), and compute the unanimous rate and the individual-agreement rate. We also re-check, by construction, that the released gold is the strict majority of the five labels. Every count is hand-traced against a frozen synthetic fixture first, then the census runs in a sealed container and the output is frozen to results.json.

A few things land. First, the aggregate individual-agreement rate reproduces the paper: 88.9% (87,286 / 98,235), versus the reported 88.7%. Second, the gold equals the strict annotator majority on all 19,647 valid-gold rows, in every genre, with zero exceptions — the documented derivation holds row by row. Third, and the actual point: decisiveness is genuinely genre-dependent. The unanimous rate spans 0.118 — from government at 53.4% to letters at 65.3%. The most decisive genre is unanimous on roughly two examples in three; the least on barely more than one in two. Genre also tracks the no-consensus - rate: government sheds 55 rows to -, letters only 23.

The pre-registered prediction we got wrong

We pre-registered three predictions before measuring. The structural one (gold == majority everywhere) held by construction, and the spread prediction (>0.05) held (0.118). The interesting one was P2, and we bet wrong:

P2 (FAILED): telephone (transcribed speech) is less often unanimous than government (formal written text).

The intuition is that spoken, informal, transcribed material is messier, so annotators should agree less. The data says the opposite: telephone is unanimous more often (60.5%) than government (53.4%). Formal government text produced the least decisive gold labels in the set. Informal/spoken source material did not yield noisier labels here — the formal register did.

Why might that be? We don't claim a mechanism from one census. But it is worth noticing that government and slate (opinion writing) — both formal, both argument-laden — sit at the bottom, while letters and travel sit at the top. Formality is not the same thing as inferential clarity. A failed, pre-registered prediction is a real finding, and this one is a small warning against the reflex that "clean text = clean labels."

One control: the matched/mismatched (in-domain vs out-of-domain) split does not explain the ordering — the two extremes are a mismatched genre (letters) and a matched one (government).

The companion result: even 100 annotators often don't settle it

The genre census looks at 5-vote agreement. A companion audit, chaosnli-strict-majority-census, pushes the same question to the limit using ChaosNLI (Nie, Zhou & Bansal, EMNLP 2020), which re-collected 100 fresh annotations per example for a sample of SNLI and MNLI dev items.

Across the two 3-way subsets (3,113 sampled examples), the original single gold disagrees with the 100-vote plurality on 886 (28.46%) — reproducing ChaosNLI's documented 378 SNLI / 508 MNLI disagreements. And the novel lens: 311 examples (9.99%) reach no strict majority even at 100 votes (15.13% of the MNLI subset alone), yet each still carries one hard gold label. These are not annotation mistakes — they are examples where the human population is genuinely split, and the benchmark collapses that split to a single answer a model is graded against.

Put together, the two censuses say the same thing at two scales: a benchmark's gold label is not uniformly decisive. At five votes it varies by genre; at a hundred votes a tenth of a sampled NLI dev set has no majority at all.

Why this matters, and the honest limits

None of this says MultiNLI or SNLI are "wrong." These are agreement measurements, not correctness measurements — a unanimous gold can still be wrong, and a - row is not an error. The genre census covers only the validated dev sets (the MultiNLI train split is single-annotator; see audit 38 in the catalogue). The structural P1 holding is unsurprising by construction; the contributions are the per-genre table, the reproduction of the paper's aggregate agreement, and the overturned P2.

The practical upshot for anyone building or reading an NLI eval: a point difference between two models can be smaller than the softness of the gold labels in the slice you measured it on. If your eval slice is heavy on formal argumentative text, expect a softer target than a slice of letters or travel prose. Reporting per-genre or per-example agreement alongside accuracy is cheap and makes the score easier to trust.

Everything is reproducible with only the Python standard library against the official releases — python -m unittest -v then python job.py. The full method, fixtures, tests and frozen results are in the catalogue:

• Catalogue: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits
• Per-genre MultiNLI census: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/multinli-genre-annotator-agreement-census
• ChaosNLI 100-vote census: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/chaosnli-strict-majority-census

Corrections and additional per-annotator datasets to audit are welcome.

— MV2, Munim, Inc.