相同輸出,不同參考標註:衡量參考選擇如何改變多語言基準分數
Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score
這項研究在一項涵蓋六種語言、針對個人身份識別資訊的基準測試中,量度更換參考標註對固定系統輸出評分的影響。更換參考標註後,兩個輸出的 F1 分數分別變動 4.95 和 2.00 點,最差語言變動 7.55 點;兩個輸出的配對交互效應為 +2.95 點。研究將差異追溯至未記錄的彙總預設,並展示如何用既有標註紀錄計算參考敏感度區間,建議把它與分數一併報告。
Published on Oct 2
Authors:
,
Abstract
A benchmark score compares a system output against a reference, and methodological attention falls almost entirely on the first term. We measure the second. The retained annotation record of a six-language benchmark for personally identifiable information contains two independent annotator labellings, the aggregate shipped as gold, and a reviewer gold from independent expert re-annotation of a sample. Using it, we hold the scored output fixed and exchange the reference. The score moves by 4.95 F1 points for one output and 2.00 for the other (95% CIs [3.23, 6.27] and [0.26, 3.37]), and by 7.55 in the worst language. The comparison between two outputs moves as well: the paired interaction between reference and system is +2.95 points (CI [+1.97, +3.87]), survives correction for multiple testing, and changes one language's margin outright. The cause is an undocumented aggregation default that usually kept one annotator when adjudication did not fire, making the shipped reference a partial copy of an output being scored. We report the resulting reference-sensitivity band, show how to compute one from any retained annotation record, and argue that the quantity belongs beside the score.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2610.03825
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.03825 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.03825 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.03825 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co