跳到正文
DEV Community· ekocam·· 15 天前AI 評分61

如何如實評估 LLM 預測:Claude Opus 智能體羣組與 Polymarket 的對比結果出爐

How to score LLM forecasts honestly: our Claude Opus swarm vs Polymarket, resolved

AI 導讀

作者以四個已結算的 Polymarket 市場示範如何評估 LLM 預測,平均 Brier score 顯示市場價格(0.167)優於所有 AI 來源。評估方法包括在結果揭曉前鎖定預測、計算 Brier score 和 log score,以及逐一移除市場和進行 permutation test。測試只有四個市場、每種配置只跑一次,作者將其定位為方法示範而非模型排名。

正文

Two weeks ago we posted forecasts from a MiroFish agent swarm running on Claude Opus 5.5, from Opus, Sonnet and Haiku asked directly, and from wallet positioning data, on four live Polymarket markets. Every number was pushed to GitHub before any market resolved. All four have now resolved, and the full results are on our blog with the scoreboard on GitHub.

Short version: the market price beat every AI source. This post is about the scoring itself, because most "my model predicts X" demos skip it.

1. Lock the forecasts before the outcome

The whole exercise is worthless if the numbers can move after the fact. We committed forecasts.csv to a public repo on October 25; GitHub's event log records the push time, and the repo went public before the first market resolved. If you run your own test, a timestamped commit is the cheapest proof you can get.

2. Brier score, against a coin flip and against the price

For a binary event with forecast probability p and outcome y (1 or 0):

def brier(p: float, y: int) -> float:
    return (p - y) ** 2          # 0 is perfect, a 50/50 coin flip always scores 0.25

Mean over the four markets:

Source Brier Log score
Polymarket price at the snapshot 0.167 0.502
Swarm on Opus 5.5 (median agent) 0.203 0.581
Opus 5.5 asked once (median of 5) 0.218 0.613
Coin flip 0.250 0.693
MiroFish report agent 0.345 0.984
Wallet money on YES (share of dollars) 0.363 1.196

A coin flip is a weak baseline. The useful one is the market price at the moment you locked your forecast, because that is what you would be trading against. We store the difference per market:

skill_vs_price = brier(p_source, y) - brier(p_market, y)   # negative = beat the price

Nothing in the table has a negative average.

3. Add a log score, Brier is too forgiving

A reader on r/algotrading pointed out that Brier goes easy on confident misses. The log score does not:

import math

def log_score(p: float, y: int) -> float:
    return -math.log(p if y == 1 else 1 - p)   # lower is better; 0.693 for a coin flip

The Bank of Canada market shows the difference. MiroFish's report agent said 89% on "no change" and the bank hiked: Brier 0.79, log score 2.21. The market price at 18.7% scored 0.035 and 0.21.

4. Check how much one market decides

With four markets, one result can flip the ranking. Here it was the Bank of Canada: 19 of 34 economists in a Reuters poll expected a hike, the market priced it at 81%, and every model given the same poll sat near 50. Remove that market and the swarm, the report agent and the wallet money all come out ahead of the price.

Leave-one-out is a two-line check worth doing on any small scoreboard:

from statistics import mean

for drop in markets:
    rest = [m for m in markets if m != drop]
    print(drop, {s: mean(brier(f[s][m], y[m]) for m in rest) for s in sources})

5. Put a number on luck: a permutation test

Another reader suggested this one. Shuffle the outcomes many times and see how often a source scores as well as it did on the real ones. With a handful of markets the answer is "often", which is the point:

import random
from statistics import mean

def perm_pvalue(forecasts, outcomes, n=10_000):
    actual = mean(brier(p, y) for p, y in zip(forecasts, outcomes))
    hits = 0
    for _ in range(n):
        shuffled = random.sample(outcomes, len(outcomes))
        if mean(brier(p, y) for p, y in zip(forecasts, shuffled)) <= actual:
            hits += 1
    return hits / n

Over the resolved markets of both rounds, no source showed skill this way, not even the market price. The sample is too small, and the test says so out loud instead of leaving it to a caveat.

6. What it showed anyway

  • The Opus swarm (0.203) and one direct Opus answer (0.218) are the same forecaster within noise. One swarm run cost $14 to $24; one answer about 2.5 cents.
  • The swarm's own report agent was the most confidently wrong source. If you use a multi-agent framework, poll the agents rather than reading the summary.
  • Wallet positioning data scored both the best single forecasts (Toronto, Iran) and the worst one (Bank of Canada). It measures where money sits, not a probability, and scoring it as one shows exactly that.

What this does not show

Four markets this round, one run per configuration, hand-corrected personas. An illustration of a method, not a ranking of models. The scoreboard is one CSV that later rounds append to, so the sample can grow.


Originally published on orcalayer.com. OrcaLayer is an independent Polymarket analytics provider, not affiliated with Polymarket.

來源:DEV Community · dev.to