跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 22 天前AI 評分45

隨機抽樣是否幾乎已經足夠?BestRand 實現穩定、低成本的基準測試壓縮

Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand

AI 導讀

BestRand 是一種穩定、低成本的基準測試壓縮方法,保留隨機抽樣效率,並減少少數大幅估算誤差帶來的不穩定。它在 70+ 項文字、多模態及智能體基準測試中,平均降低高誤差風險 50.2%、估算誤差 16.4%,建構成本只需 51.3%。程式碼及數據已公開。

正文

Abstract

Benchmark compression reduces the often prohibitive time and cost of model evaluation by evaluating models on a small benchmark subset that closely matches full-benchmark results. To quantify estimation error, we derive an accurate analytical approximation to the error distribution when estimating full-benchmark scores from a subset, showing that most subsets yield small errors, while a few produce much larger errors, resulting in a long-tailed distribution. This means random sampling alone can achieve surprisingly strong performance with high probability, but remains unstable due to rare large errors. We further show that this instability is inherent to modern foundation models and persists even as more models are evaluated. Based on these findings, we introduce BestRand, a stable and low-cost benchmark compression method that preserves the efficiency of random sampling while substantially reducing its high-error tail. We validate BestRand on 70+ text, multimodal, and agent benchmarks, and further extend it to VLA evaluation. Compared with competitive methods, BestRand reduces high-error risk by 50.2% and estimation error by 16.4% on average, while requiring only 51.3% of the construction cost. It further reduces OOD estimation error by 21.8% on average across score-range, model-family, and temporal shifts. Together, our theory provides practical guidance for understanding and predicting the accuracy and reliability of benchmark compression, while BestRand enables stable and accurate compression at low cost. The code and data are publicly available onhttps://github.com/George-Ling3/BestRand.

View arXiv page View PDF GitHub Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.00883 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.00883 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.00883 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co