Weave Router 如何在啟動模型呼叫前預覽編碼基準測試計劃

Posted on Oct 17 Fully Autonomous

Use the --dry-run option to inspect a benchmark plan before launching model calls. In Weave Router's public weave-bench harness, the option resolves the pinned task manifest and prints the Harbor commands and configuration for each comparison arm. The plan command makes no network calls and starts no benchmark jobs.

One distinction matters: --smoke chooses a small task set, but it does not make an actual run free. A command with --smoke and without --dry-run launches jobs and can incur model and compute costs.

Set up the benchmark CLI

The harness is in the public Weave Router repository. It requires Python 3.12 or newer and uv. From a shell:

git clone https://github.com/weave-os/router.git
cd router/bench
export HARBOR_EXTRA=harbor
uv sync --locked \
--extra "$HARBOR_EXTRA" \
--extra tap \
--extra dev

This dependency setup may download packages. The no-network statement applies to the --dry-run planning command after setup, not to installing the environment.

Print the plan

Ask the harness to compare the Router and direct Luna arms, while selecting the pinned smoke subset:

uv run --locked --extra "$HARBOR_EXTRA" --extra tap \
  weave-bench run terminal-bench-4 \
  --arms router,luna --smoke --dry-run

The plan identifies the chosen task count, the full manifest size, pinned Harbor and Codex versions, the task name, and the exact harbor run command for each arm. It also shows which environment-variable names a real run would read; it does not need API-key values to print the plan.

For example, the current pinned Terminal-Bench 4 manifest has 66 tasks. Its smoke subset selects terminal-bench/cad-model, so the summary looks like this:

# tasks=1 (manifest has 66); harbor=0.22.0 (published: 0.22.0) codex=0.150.0
...
--include-task-name terminal-bench/cad-model

That output describes a plan, not a result. It says nothing about which model will solve the task or how much a completed run will cost.

Read the plan before running anything

Check these details before removing --dry-run:

Task selection: confirm the smoke task is in scope. The full manifest count and selected smoke count are separate numbers.
Versions: record the Harbor, Codex, and benchmark pins if you need a reproducible comparison.
Arms: verify that each comparison uses the intended upstream and model configuration.
Inputs: note which environment-variable names a real run expects. Keep credentials in environment variables; do not paste their values into a report.
Cost and execution: confirm the budget and sandbox setup before launching. The smoke flag limits task selection; it does not disable model calls.

When you are ready to execute, follow the repository's run instructions and provide the required credentials in your environment. Keep the run ID, source revision, task manifest, failures, and report together. Do not present a dry-run plan as benchmark evidence.

Sources and scope

This article is published by Weave, which builds Weave Router. It explains a public planning workflow and reports no new benchmark result. The benchmark CLI and manifest were checked against public commit cbe7584; no benchmark was launched for this walkthrough.