MiroFish 智能體羣以 Claude Opus 5.5 測試四個市場，花 $98；單次 2.5 美仙呼叫得出相同預測

Multi-agent swarms are a popular way to get "more than one model" out of an LLM: generate dozens of personas from your documents, let them post and argue on a simulated social network, then read off what the crowd thinks. We tested one, the open-source MiroFish engine, as a forecaster on four live Polymarket markets, with Claude Opus 5.5 behind every agent. The full results are on our blog, and every forecast, prompt, seed file and persona correction is on GitHub.

Short version: the swarm landed 0 to 3 points from one direct Opus answer on all four markets, at 500 to 900 times the cost. This post is about how to evaluate an agent swarm so you can see that, and four things we would check in any multi-agent setup.

1. Always run the single-call baseline

The most important number in the whole test was not a swarm output. It was the same model asked the same question directly, with the same seed files and instructions, five times, median taken:

Without the baseline, a swarm result looks like an emergent crowd opinion. With it, you can see the crowd is the base model's view of the evidence, re-voiced 34 to 56 times. Sonnet showed the same pattern (50 vs 52, 49.5 vs 50).

In our first round, repeat runs of the same setup moved the median by up to 10 points, so a 0 to 3 point gap is noise. One run per configuration is a real limitation; we spent the budget on Opus instead of repeats.

Why it happens: every persona is written by the same model from the same documents. The agents share the same facts, framing and blind spots. Diversity of roles is not diversity of information.

2. Poll the agents, don't trust the summary

MiroFish ends a run with a report agent that writes a summary and its own probability. On the Bank of Canada market, three report agents said 89 to 93, and the Opus report had a chapter titled "Agent Consensus: Banks and Economists Converge on 'No Move'".

We asked each agent directly instead. Median: 45. Only 13 of 34 agents put YES above 50.

The cause was mundane: the report agent tries to interview the agents, but by then the simulation had shut down, so it wrote from the knowledge graph built from the seed files and never saw the agents' answers. If a framework gives you a "consensus" summary, check that it actually read the outputs it claims to summarize.

3. Validate generated personas against the source

Before each run we compared every generated persona with the seed files, first with a second model, then by hand. Opus 5.5 still produced about 130 factual errors across four markets. Several hit the exact fact a market resolves on:

• six Israeli personas moved the election to November;
• three reversed a poll result (the seeds had Eisenkot ahead of Netanyahu 49 to 41);
• several said the Bank of Canada had just hiked, mixing it up with the Fed and the ECB;
• one poll's district sub-result became its overall result in five personas;
• the generator created an account for a mayoral campaign of a person who wasn't running.

It also liked to end the simulation setup with a verdict ("the narrative should drift toward cautious confidence that the pause holds"). That is the generator steering the outcome before any agent speaks. We removed those sentences and 11 opening posts with wrong facts or ready-made conclusions, and published originals and fixes.

One way to make this routine: for every persona, extract its factual claims as a list, check each against the seed files, and flag anything not supported. A cheaper model is fine for the first pass; a human for anything that touches the resolution criterion.

4. Meter the real cost, and cache

Every agent re-sends its persona, tool list and history on every turn, so cost grows fast with agents times rounds.

Prompt caching cut the Opus bill by about 40%; without it a run would have been $23 to $40. The whole round, with preparation and persona checks, was $98.56.

Also worth doing: lock the forecasts publicly

A forecasting test is only honest if the numbers are fixed before the outcome. We pushed forecasts.csv to GitHub before any market resolved, and GitHub's event log records the push time. The scorecard (Brier score and absolute error per source, including the market price) comes after the last market closes on October 31.

What this does not show

Four markets this round, eight across two rounds: an illustration, not a ranking. One run per configuration. Hand-corrected personas, so an uncorrected swarm might behave differently. Accuracy comes after resolution.

The takeaway so far: putting a frontier model behind every agent did not make the swarm a different forecaster from the model it runs on. If you want what the model thinks, ask it once.

Originally published on orcalayer.com. OrcaLayer is an independent Polymarket analytics provider, not affiliated with Polymarket.