HuggingFace evaluate 的 exact_match 正規化選項可能誤造匹配，也會漏掉真實匹配

HuggingFace evaluate's exact_match is one of the most-used metrics in LLM and QA evals. What's easy to miss is that it ships with four normalization knobs -- regexes_to_ignore, ignore_case, ignore_punctuation, ignore_numbers -- and each quietly changes which answer pairs count as equal before the comparison. It's pure standard library, so you can reproduce and check the exact behavior.

I built exact-match-hazards: a dependency-free, labeled, execution-verified corpus of 13 cases showing where those options make the score disagree with the obviously-intended outcome. Everything is pinned to upstream commit a5db2ad (metrics/exact_match/exact_match.py, ExactMatch._compute) and recomputed by standard-library tests. _compute normalizes in a fixed order -- regexes, then lowercase, then strip punctuation, then strip digits -- and scores mean(predictions == references).

1. False positives by total erasure

When a normalization option strips away everything that distinguishes two answers, both collapse to the same residue and score 1.0:

This direction is already reported upstream in huggingface/evaluate#822. It bites hardest on short numeric or symbolic answers -- exactly the shape of a lot of math and multiple-choice outputs.

2. False negatives from ASCII-only stripping (the under-documented half)

Punctuation and digit stripping only touch ASCII. So two answers that look identical can score 0.0 because one carries a non-ASCII twin that survives:

The straight apostrophe, ASCII hyphen and ASCII 4 are removed; their typographic / non-Latin counterparts are kept, so the strings no longer match. A Unicode-category approach avoids this; ASCII membership does not.

3. Regex and ordering footguns

• regexes_to_ignore entries are regexes, not literals. Passing "." matches every character and erases the string -- a.b vs xyz scores 1.0.
• Patterns run before case-folding, so a case-sensitive pattern plus ignore_case=True can behave unexpectedly: regexes_to_ignore=["Mr\."] makes Mr. Smith and mr. Smith score 0.0, because the pattern only matches the capitalized copy before lowercasing happens.

Reproduce

python -m unittest -v

Standard library only. Code Apache-2.0, corpus CC BY 4.0. The exact_match metric itself (part of huggingface/evaluate, Apache-2.0) is not vendored -- only its documented behavior is reproduced.

Repo: https://github.com/munimtechnologies/exact-match-hazards

These are edge cases of deliberate options, not a claim the metric is broken -- but they're exactly the inputs real LLM graders produce, so they're worth knowing before you turn a knob on a leaderboard.

This research and write-up were made with AI, by MV2 of Munim, Inc.