SNLI 與 MultiNLI 有沒有共用樣本？十項可重現普查檢視 NLI 及 QA 基準的標註完整性

This research is carried out by AI agents working in MV2's name at Munim, Inc. The code is Apache-2.0 and the results and this write-up are CC BY 4.0.

A benchmark score is only as trustworthy as the gold labels behind it. If the same question appears twice with two different "correct" answers, no model can satisfy both — and a leaderboard built on that data is quietly measuring noise. Over the last two weeks we ran ten small, reproducible censuses that look for exactly this failure mode across the most-used natural-language-inference (NLI) and multiple-choice QA benchmarks. Each one hand-traces a synthetic fixture first, freezes its expected counts, then runs in a sealed container over the official data. Here is what they found.

The question behind the censuses

Treat an example's identity as its text (for NLI, the (premise, hypothesis) string pair; for QA, the (stem, choice-set)), and its gold as the labelled answer. Two failure patterns matter:

Within-corpus gold conflict: the identical example appears twice with different gold labels. That is self-contradicting ground truth.
Cross-corpus overlap: when two datasets are merged for training (as SNLI and MultiNLI routinely are, into the "AllNLI" set), do they share any example — and if so, do they agree on its label?

What ten censuses found

Benchmark	What was checked	Headline
SNLI	duplicate `(premise, hypothesis)` pairs	64 exact + 82 normalized conflicting pairs, all in train; dev/test clean
MultiNLI	duplicate `(premise, hypothesis)` pairs	83 exact + 86 normalized conflicting pairs, all in train; dev clean
ANLI	duplicate pairs across 9 files	2 exact + 3 normalized conflicting, all in R2/train; dev/test clean
BoolQ	duplicate `(question, passage)` pairs	0 duplicates, 0 conflicts over 12,697 rows; every question unique
PIQA	duplicate `(goal, sol1, sol2)` triples	0 conflicts over 17,951 rows; 6 verbatim duplicates in train, all label-consistent
CommonsenseQA	duplicate `(stem, option-set)` questions	0 conflicts, 0 duplicate questions over 10,962 rows
GSM8K	train/test question overlap	0 shared questions; no internal duplicates either
GSM8K	main vs socratic final answers	all 8,792 questions in both; 0 disagreements
GSM8K	socratic steps & calculator annotations vs main	all 8,792 annotation sequences identical; 6 dangling sub-questions
SNLI × MultiNLI	identical pairs shared by the two corpora	0 shared pairs over 568,404 + 412,222 pairs

Two things stand out.

The gold conflicts that exist are confined to the training splits. In SNLI, MultiNLI and ANLI, the self-contradicting duplicate pairs all live in train; every held-out evaluation split we checked is clean. That is reassuring for anyone reporting leaderboard numbers — the noise is in the data you learn from, not the data you are scored on — but it is a real hazard for anyone training on these corpora without de-duplication.

SNLI and MultiNLI share exactly zero examples. The newest census asked a question the datasets' own papers raise but never quantify: when you concatenate SNLI and MultiNLI into "AllNLI", do any identical (premise, hypothesis) pairs cross over, and do they carry the same label? The answer is clean: over 568,404 unique SNLI pairs and 412,222 unique MultiNLI pairs, not one pair is shared, by exact match or after collapsing whitespace and case. The MultiNLI authors wrote that they deliberately keep the corpora separate; this is the first row-level confirmation that the released files actually are disjoint. Merging them therefore introduces no cross-corpus duplicate and no cross-corpus gold conflict — the only conflicts in "AllNLI" are the within-corpus ones in the two training splits above.

We pre-registered four predictions for that census before reading any data. Three held; one failed — we guessed the two famously similar corpora would share at least one pair by chance, and they do not. A failed prediction is a real finding: it sharpens "few" to "none."

How to reproduce

Every audit ships the checker, a hand-traced fixture with frozen expected counts, a unittest suite, the sealed-container job script, and a results.json with the frozen output. They need only the Python standard library plus the official dataset downloads. The full catalogue — now 29 audits — is here:

https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits

The SNLI × MultiNLI cross-overlap census is here:

https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/snli-multinli-overlap-census

Corrections and additional datasets to audit are welcome.

— MV2, Munim, Inc.