Qwen3.8-27B 在 LiveCodeBench 表現與 Claude Fable 5 相當，成本約為後者的 78%而非 9%

Qwen3.8-27B, an open-weight model served locally with vLLM, scored 92.4% pass@1 on the 100 most recent hard LiveCodeBench problems. Claude Fable 5, called once, scored 90.4%. Nobody fine-tuned anything. The GVS5H repo gets there by starting fresh Qwen3.8-27B instances over and over and making them coordinate through a handful of files on disk.

That's the result going around, usually with "at 9% of the cost" attached. I read the paper source in the repo. The 27B did match Fable 5. It didn't beat it in any statistical sense, it wasn't cheap, and the 9% figure belongs to a different model. With the harness, the 27B costs about 78% of what Fable 5 does at list prices.

Qwen3.8-27B vs Claude Fable 5 on LiveCodeBench: the actual table

The main table (paper/tab-4new-5pass.tex) covers LCB-100: the 100 most recent problems in the hard split of LiveCodeBench release_v6, with a 128k output cap, reasoning on and five passes per arm. "Manager" means the GVS5H scaffold.

All five models that ran both arms gain from the manager, and the weaker the single call, the bigger the gain. The authors put it this way: "the smaller the model, the more the scaffold has to add."

The "9% of the cost" figure is for Flash-Next, not the 27B

This is where the reposts get it wrong. The abstract says: "Orchestrated Qwen3.8 Flash Next scores 93.0% against Fable 5's 90.4% at 9% of the cost, while the smaller Qwen3.8-27B reaches 92.4%." The 9% applies to Flash-Next only.

Here's the cost table (fig-cost-tables.tex), priced at list rates per pass over all 100 problems:

$5.76 / $61.11 is about 9%. The 27B with the manager comes to $47.82, which is roughly 78% of Fable 5. The paper calls this "a slightly lower price", which is fair. The 27B talks a lot while it reasons: its single call alone emits 7.42M output tokens per pass, against 1.20M for Fable 5.

Those dollar figures are OpenRouter list rates ($0.21 in / $2.55 out per MTok for the 27B). On your own GPU the marginal cost is electricity and waiting time. On HN, badsectoracula reports about 50 tok/s for Qwen 3.8 27B on an RX 7900 XTX (Unsloth Q4 quant, llama.cpp). At that speed the manager arm's 18.6M output tokens per pass take about 100 hours of generation for one run of 100 problems on a single card, assuming no batching. Someone in the thread asked "But in 10x the time?" If you're running this at home, that's the number I'd want answered first.

Does Qwen3.8-27B beat Fable 5? Statistically, no

The +2.0 points don't hold up, and to its credit the paper says so. The p-value for Qwen3.8-27B manager vs Fable 5 single is 0.88 after Holm correction, and the one-sided 95% lower bound on the difference is -0.8 points. You can say it matches Fable 5. You can't say it beats it.

Six caveats, in order of how much they bother me

1. The 27B baseline was hurt by the token cap

The single call was generated at 250k and then replayed at 128k so it matched the manager arm. At 128k, 150 of 500 single calls (30.0%) hit the cap and 35 (7.0%) produced no code at all. At 250k the single call scores 70.0, not 66.8 (tab-qwen-caps.tex). Part of the "+25.6" comes from the manager splitting long reasoning across fresh contexts so nothing gets cut off. That's a real benefit of the scaffold, though anyone reading "+25.6" as "the model got smarter" is reading too much into it.

2. The manager arm gets to run tests and the single call doesn't

The v2 scaffold has a verifier step that runs each candidate against the problem's public stdin tests and feeds the first failure back to the manager. That covers 73 of the 100 problems. On HN, sams99 argued that this adds Python execution to a comparison that was supposed to be zero-tools. I agree, and I'd have liked to see a single call with one test-and-repair round as the baseline.

3. Up to 10 rounds of calls per problem against one call

Both arms ran five passes, so that part is even. Inside each pass, though, the manager arm gets a plan, a brainstorm, up to 10 manager→worker cycles, a finalizer and cut-off summarizers per problem, while Fable 5 gets one call. Fable 5 was run "single-only due to its high cost", so nobody knows what Fable 5 with a manager would score.

4. The Opus comparison uses a different setup

Opus-5 (88 → 95 with the manager) ran on the original scaffold through OpenRouter, for one pass. The original scaffold has 4 cycles instead of 10 and no verifier. The paper does say the 27B reaches "Opus-5's [range] in the OpenRouter-served set", but its own OpenRouter table notes the two sets "are not pooled" because the serving path and scaffold version differ. Treat "27B ≈ Opus" as a loose cross-table comparison.

5. Gains aren't universal

On the OpenRouter set, Qwen3.6-35B with reasoning off went from 35 → 27 at 128k and 28.2 → 27.4 at 16k. Kimi-K3 with reasoning on went 86 → 86. The failure case in the paper shows how this happens: the manager arm's ideation stage found the same optimization the single call had used, talked itself out of it as too error-prone, and committed to an asymptotically slower plan.

6. The authors ran their own benchmark and fixed their own grader

They patched two evaluator issues (non-unique reference answers on four problems, and a sys.stdin mock whose sys.stdin.buffer.readline() returned the first line on every call). Both fixes look reasonable to me. Still, nobody independent has reproduced this. Fable 5 also returned stop_reason=refusal on 6 of 500 problem-passes, which costs it 0–1.2 points.

On contamination: HN commenters called the benchmark old and saturated. The paper picks the 100 most recent hard problems by contest date specifically to limit that, but a model released after those contests could still have seen them.

The ledger pattern you can copy into your own agent loop

None of the above affects the part worth taking home. Every call starts with an empty context and reads state only from these files:

<ws>/task.md        the problem statement
<ws>/plan.md        the manager's plan (capped at 4,000 chars)
<ws>/notes.md       curated findings (capped at 8,000 chars, rewritten, not appended)
<ws>/solution.py    the current solution
<ws>/tasks.json     debug mirror of the task list

The loop: the manager writes a plan and seed tasks, a worker brainstorms into notes.md, then the manager picks one next task, a fresh worker executes it and rewrites solution.py plus notes.md, and the public tests run. Repeat until the manager says done, it reissues the same task, or 10 rounds pass. Temperatures are 0.3 for the plan, 0.4 for the brainstorm and 0.2 for execution.

To try the authors' harness, here is the command from the README (the example uses an OpenAI model; swap the MULTIAGENT_MODEL provider prefix for your own endpoint):

cd codebase/v2-current
export OPENAI_API_KEY=...

LCB_RELEASE=release_v6 \
ESCALATION_CLOUD_MAX_TOKENS=128000 \
ESCALATION_CLOUD_TIMEOUT=7200 \
MULTIAGENT_MODEL=openai:gpt-5.6-terra \
uv run --no-project --python 3.12 --with 'datasets<4' --with numpy --with anthropic \
  python escalation/run_bench.py --engine multiagent --only lcb --lcb 100 --parallel 8

--engine single gives you the baseline. Results go to runs/results.json.

If you already run a local agent, here's what to steal:

• Cap the notes file and make workers rewrite it instead of appending. Unbounded notes are how agent loops rot.
• Give each worker one task plus the ledger, never the full transcript.
• Treat a failing public test as ground truth that overrides the manager's "done".
• Add a no-progress guard. If the manager hands out the same task twice, stop.
• The paper's own suggestion: occasionally spawn a worker that sees only task.md. Inherited notes anchor every worker to an early wrong idea.

My take: if you run a 27B locally, GVS5H is good news. Structure plus a test runner got a mid-size model to frontier-level pass@1 on hard problems. I wouldn't read it as evidence that a 27B is as smart as Fable 5, and I'd stop quoting the 9% figure next to the 27B.

So is a 27B looping through up to 10 manager→worker rounds against one call of Fable 5 a fair fight? Or is that exactly the point, since your local GPU doesn't bill per token? Tell me in the comments, especially if you've run a ledger-style loop on your own hardware and watched it make things worse.

Sources

• GVS5H repo and paper source: https://github.com/slee-persis/GVS5H (paper/paper_latest.tex, tab-4new-5pass.tex, fig-cost-tables.tex, tab-qwen-caps.tex, tab-openrouter.tex)
• Hacker News discussion: https://news.ycombinator.com/item?id=50228167