跳到正文
Hacker News 熱門（buzzing.cc 中文翻譯）·· 3 小時前AI 評分67

GVS5H 研究顯示編排後 Qwen3.8-27B pass@1 升至 92.4%

五款 Qwen3.8-27B 型号与 Fable 5 兼容，使用 GVS5H 线束

AI 導讀

GVS5H 提出一種免訓練的零樣本自我編排方法，讓同一模型的多個新實例透過共享檔案系統分解問題並協作解題。在 11 款開放及封閉權重模型的 100 道最新高難度 LiveCodeBench 題目上，該方法在固定後端帶來最高 25.6 個百分點提升；本地部署的開放權重 Qwen3.8-27B pass@1 由 66.8% 升至 92.4%，但部分模型沒有改善或表現變差。

正文

GVS5H: Gorilla vs. 5 Humans

Five boys standing on one another's shoulders, together as tall as a single gorilla beside them

slee-persis%2FGVS5H | Trendshift

Results

Accuracy on LiveCodeBench, single call versus with manager, for seven models against Claude Fable 5's single-call score

Opus 5 ran on an earlier version of the scaffold, in a single pass served through OpenRouter rather than five on a pinned backend.

Average cost per task against accuracy, single call versus with manager, on a log-scale cost axis

What one pass costs — LCB-100, 5 passes, single call vs manager, against Fable 5

Abstract. Frontier coding performance is typically bought with larger proprietary models at high cost. We introduce ledger-based zero-shot self-orchestration, a training-free method in which fresh instances of one model decompose problems and coordinate through a shared filesystem holding a plan, notes and current solution. Across eleven open and closed-weight models on the 100 latest hard LiveCodeBench problems, the method yields gains of up to 25.6 percentage points on pinned backends and offers two routes to frontier-level accuracy. Orchestrated Qwen3.8 Flash Next reaches 93.0% pass@1 against Fable 5's 90.4% at 9% of the cost, and locally served, open-weight Qwen3.8-27B rises from 66.8% to 92.4%. Gains are not universal: some models are unchanged or worse. Transcript analysis attributes the gain to decomposition and persistent context. Inference-time organization can approach frontier coding accuracy at a fraction of the cost, or slightly exceed it on self-hostable weights.

— the paper

Running the code

Needs uv and an API key for the model you want to test.

cd codebase/v2-current
export OPENAI_API_KEY=...

LCB_RELEASE=release_v6 \
ESCALATION_CLOUD_MAX_TOKENS=128000 \
ESCALATION_CLOUD_TIMEOUT=7200 \
MULTIAGENT_MODEL=openai:gpt-5.6-terra \
uv run --no-project --python 3.12 --with 'datasets<4' --with numpy --with anthropic \
  python escalation/run_bench.py --engine multiagent --only lcb --lcb 100 --parallel 8
  • --engine multiagent runs the manager; --engine single is the one-call baseline.
  • Other models: anthropic:<model>, dashscope:<model>, openrouter:<model>, each with its own *_API_KEY.
  • The pass@1 score prints at the end. Results are written to runs/results.json, workspaces to runs/ws/.

License

Code is under the MIT License. The paper, figures and run data are under CC BY 4.0. The LiveCodeBench fork, the benchmark problem statements and the LaTeX template files keep their own licenses. See NOTICE.md for which license covers which path.

來源：Hacker News 熱門（buzzing.cc 中文翻譯） · github.com

#論文/研究#編碼#Agent#GitHub