3,671 次預測看似理想，我的體育博彩模型仍未勝過莊家
My betting model looked great for 3,671 predictions. It never beat the bookmaker.
作者記錄 16 天、共 3,671 次體育博彩預測，發現自己的模型在可與莊家比較的 510 次預測中，表現與市場無法區分。在這 510 次預測中，模型的 Brier score 為 0.1874，莊家為 0.1863，配對差值為 +0.00105，95 % 信賴區間為 [−0.00063, +0.00276]。
For 16 days I logged every probability my sports-betting model announced before a match, and what actually happened afterwards: 3,671 settled predictions.
By the usual numbers, it looked good:
Brier score 0.1629 (lower is better)
30.8 % better than predicting the base rate every time
Calibration error (ECE) of 3.0 %: on average it announced 37.8 %, and 38.2 % happened
I would have told you it worked. It didn't.
Better than what?
A Brier score on its own mostly measures how hard the questions are. The only question that matters for betting is: is it better than the bookmaker's own odds?
On the 510 predictions where I had the bookmaker's probability (vig removed) for the very same outcome:
Brier
My model 0.1874
The bookmaker 0.1863
The paired difference is +0.00105, with a 95 % confidence interval of [−0.00063, +0.00276]. It leans slightly against me, and it is too small to tell apart from zero. Verdict: indistinguishable from the market. A model that only matches the bookmaker cannot make money against the bookmaker's margin.
The number that would have saved me weeks
The most useful measurement needs no match results at all: how close my probabilities were to the bookmaker's.
median gap: 1.0 point
correlation: 0.9941
My model wasn't predicting football. It was copying the odds it was fed and nudging them a little. A model anchored this tightly cannot beat its reference, by construction, and that was measurable on day one, before a single match was played.
The tool
I turned the measurement into an open-source harness, eval-pack. Zero dependencies, plain TypeScript on Node ≥ 22.6. Its main rule: no verdict without a reference. If you don't tell it what your system must beat, it refuses to grade it.
Reproduce every number in this post:
git clone https://github.com/ewan147/llm-eval-public
cd llm-eval-public/pack
node cli.ts run forecast
The data is real and anonymised (no team, player, operator or person names, no amounts).
It works the same way for LLM systems: a classifier, a RAG pipeline, a judge, an agent. "Is the new model better?" is the same question as "does my model beat the bookmaker?": better than what, on how many paired cases, and with what confidence interval.
來源：DEV Community · dev.to