Squoosh Bench 評估 UX-1 在 318 項歷史網站 A/B 測試中的表現
Squoosh Bench: Evaluating UX-1 on 318 Historical Website A/B Tests
Squoosh Bench 評估 UX-1 對歷史網站 A/B 測試結果的預測,在 318 項測試中正確判定 267 項,準確率為 84.0%。UX-1 以人類瀏覽數據微調,每個測試組模擬 1,000 名購物者;Qwen3 Next 80B 正確判定 196/318 項,準確率為 61.6%。UX-1 在 126 項測試中預測變體勝出,其中 118 項實際勝出、4 項落敗,另有 4 項結果不明確。
Published on Nov 18
Authors:
,
,
,
Abstract
Experimentation teams must decide which proposed website changes to test with live traffic. We evaluated whether forecasts from simulated shoppers could inform that choice. We screened 412 historical website A/B tests and evaluated four systems on the 318 that ran longer than 21 days. Each system predicted an original win, a variant win, or an inconclusive result. Page roles were masked and recorded outcomes were withheld during inference. Squoosh UX-1, a model fine-tuned on human browsing data, simulated 1,000 shoppers per arm and correctly classified 267 of 318 outcomes (84.0%, 95% Wilson interval 79.5-87.6%, treating experiments as independent). Qwen3 Next 80B, the strongest general-purpose configuration on outcome accuracy, correctly classified 196 of 318 (61.6%). UX-1 favored the variant in 126 experiments. In the live tests, 118 of those variants won (93.7%), four lost, and four had inconclusive results. The remaining 192 experiments included 19 additional variant wins.
View arXiv page View PDF Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.25200 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.25200 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.25200 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co