跳到正文
DEV Community· bayze-mcp·· 16 天前AI 評分58

bayze-mcp 團隊以兩個 AI 辯論、第三個 AI 仲裁,六輪測試修正 7 個問題

We Made Two AIs Debate and Hired a Third as the Arbiter — 6 Test Rounds, 7 Bugs Fixed

AI 導讀

bayze-mcp 團隊以兩個 AI 辯論、第三個 AI 仲裁測試多模型協作模式,經六輪實測修正 7 個問題。測試揭示上下文被編排器丟棄、輸出長度限制不一致、雙方立場重合及模型捏造數據等問題;文章比較五種協作模式和成本,認為任務與模式匹配比模型質素更重要。在雲端遷移案例中,仲裁結論是不應讓 8 人初創公司在 12 個月內全面遷移;兩個模型給出的所有金額數字均屬捏造,並已被仲裁者剔除。

正文

Three months ago bayze-mcp was a single waterfall: model A drafts, model B reviews, model C revises. Today it ships five multi-model collaboration patterns — and the most interesting one, debate, only works because we broke it six times with real runs and fixed what broke. This post is the full debugging log, costs included.

The five patterns, with real costs

Pattern Mechanism Measured cost Use when
Cross-vendor review A drafts → B structured-review gate → C revises ~$0.001/run Default quality tier
Parallel + judge N models answer → judge picks by hard constraints $0.0013/run Tasks with length/format caps
MoA aggregation N models answer → synthesizer merges the best of each $0.006/run Open-ended generation
Revision loop Review bounces → revise → re-review → escalate to a stronger model when exhausted $0.002/escalation Unstable first drafts
Debate Two debaters open independently → K rebuttal rounds → arbiter verdict $0.02–0.04/run Analysis / decisions / risk assessment

The single most useful finding: pattern-task fit beats model quality. With a hard length cap, MoA aggregation overshot by +59% and +74% in two runs (rewrite steps systematically lengthen text — same family as the 1.66× inflation we measured in revision steps), while a judge picking between candidates naturally respects the cap because it just picks the shorter one. For open-ended writing the judge can only hand you "the better of two drafts"; only synthesis takes the best of both.

Nobody counts their own words

Five samples across six rounds:

  • Qwen-plus self-reported "exactly 10 words" — actual: 14
  • Qwen-plus self-reported "about 580 characters" — actual: 799
  • Claude haiku under-reported by 1–38 chars in earlier runs

No model's self-count is trustworthy. So bayze treats length like memory safety: the engine counts, deterministically — every output passes max_chars / max_words checks, and the model's opinion is never consulted.

Six rounds of debate testing: a bug-finding story

We stress-tested strategy: debate with a real question: "Should an 8-person startup migrate off a cloud vendor's full stack within 12 months?"

Rounds 1–2, two invisible bugs. The test agent complained "my constraints are being ignored." They were — but not by the models: the context field was accepted by the MCP layer and silently dropped by the orchestrator. Separately, maxOutputTokens: 2000 allowed ~3000 characters of output while a max_chars: 2000 gate rejected at 2000 — the engine was permitting output it was guaranteed to reject.

Round 3, the debate actually worked. Rebuttals quoted each other point by point. One side caught the other promising "RPO < 5s" in its opening while admitting "primary failover takes 3.2 hours" in the same document. Cross-model adversarial review genuinely surfaces blind spots a single model has. Then the arbiter wrote 786 characters against a 600-character gate and the entire verdict was dropped — that one arbiter call cost $0.021, 90% of the run.

Rounds 4–5, engineering answers to "vibes" problems. Models treated "opening / rebuttal / verdict" discipline as their assignment and performed the whole debate solo — fixed with a system prompt scoping each side to its own position. A token cap derived at 1.4 chars/token got bypassed at 2.43 chars/token on mixed Chinese-English Claude output — fixed by self-calibration: measure the actual chars/token ratio on the first call, use it to clamp a compress-retry. And qwen fabricated "we audited this team's real invoices (redacted)" plus fake benchmarks and Git-history claims — a workflow has no access to any of that. Fixed with an arbiter rule: unverifiable quantitative claims are discounted, suspected fabrications are called out in the verdict.

Round 6, the last design flaw. Both models ended up on the same side ("worth migrating"). Explicitly assigning sides did nothing, because both steps received the same input — there was no injection point for stances. Emergent disagreement works on open analysis questions; on yes/no questions both models anchor to the same conclusion. Fixed with optional stances: [pro, con], with one rule baked into the prompt: argue your assigned side as hard as you honestly can, but never fabricate evidence for it.

What the arbiter actually concluded

Condensed from two debates plus verdicts:

Don't do a 12-month full migration. An 8-person team needs 0.5–1 FTE of SRE capacity to self-host; the savings don't cover half a person. AI services are the last thing to move. The correct use of 12 months is making exit cheap (containers, S3/Postgres-compatible interfaces, moving traffic-heavy spend first) — then using "we can leave" as negotiating leverage.

Caveat that matters more than the conclusion: every dollar figure both models produced was fabricated (the arbiter flagged and excluded them). What survives is the decision framework and the risk checklist. That's the meta-lesson of all six rounds: model quantification is not directly trustworthy — cross-model adversarial review plus an arbiter discount just makes the untrustworthiness visible.

Try it

npx -y bayze-mcp@0.14.0 mcp --config your-config.yaml

MIT, local-first, keys never leave your environment, 193 automated tests. If you're orchestrating multiple models, I'd genuinely like to hear what yours fabricates — we keep a list.

來源:DEV Community · dev.to