MultiO 研究指出,AI 模型獨有主張即使大致正確,也可能因誇大而誤導
The most dangerous AI claims aren’t wrong — they’re overstated
MultiO 的模型分歧研究發現,AI 模型獨有主張即使方向大致正確,也可能因機制、措辭強度或數字誇大而誤導。
Most discussion of AI errors sorts claims into two buckets: right and wrong. Our divergence study needed a third bucket — and it turned out to be the one I now watch for most.
We sent the same 24 questions, unchanged, to five mid-tier models — GPT, Claude, Gemini, Grok and Llama — and hand-checked all 14 sole-source claims: claims exactly one judged model made that nobody else did. The verdicts split three ways, not two: 8 were the highest-value line in their set, 3 were wrong or invented, and 3 were overstated or mixed — directionally plausible, but wrong in the mechanism, the strength, or one number.
Three examples, one of each flavour:
Failed-payment retries (overstated mechanism): only one model claimed that ignoring issuer signals gets a merchant ID blacklisted and causes hard declines. Hand-checked, that was overstated and causally backwards — retry discipline genuinely matters, but the blacklisting mechanism as stated does not hold up. The advice sounded operational and specific, which is exactly why it would survive a casual read.
Minivan vs 3-row SUV (overstated strength): one model’s cost, fuel and safety comparison was directionally right — minivans do tend to cost less — but its claim that hybrid minivans get “far better” fuel economy is loose now that three-row SUVs have hybrids too. One word, “far”, quietly upgraded a tendency into a gap.
First-marathon plan (mixed): only one model gave concrete specifics — 4–5 runs a week, a 30–40-mile peak, an 18–20-mile long run, a 2–3-week taper, 60–90 g of carbs per hour. The structure is standard and was the most actionable line in its set, but the carb figure is aggressive for a first-timer. Mostly the best answer available, with one number to discount.
That middle bucket is harder to handle than an outright invention. A wrong claim can be falsified and discarded. An overstated one passes a fact-check on its direction, keeps its useful core, and smuggles the exaggeration through with it — into a pricing decision, a purchase, a training plan. And a single-model workflow gives you no signal it is even a lone claim: across the study, 15 of 24 questions contained a lone-model element, and a single chat window never flags that its model is alone.
My practical rule now has a middle branch: if one model adds a fact nobody else mentioned, check the fact; if it dissents, read its reasons; and if a lone claim checks out as broadly right, still discount its strongest word and its most precise number first — that is where the overstatement lives.
Full study, method and caveats (including that one judge was also a panel member, the rotating judging sets, and a memory-bleed incident we found and disclosed): https://multio.ai/research/model-divergence
Disclosure: I build MultiO (multio.ai) — One subscription. Every top AI model… Auto sends each message to the most cost-efficient model that can handle it. Compare asks up to three models side by side, and every message shows its exact token charge on a receipt. Free trial: 3,000,000 tokens, no card required.
來源:DEV Community · dev.to