部署 AREX-2 前要檢查甚麼：相同 config.json 為何不代表 serve 命令相同

When a fine-tune declares the same architecture class as a model you already serve (run behind an API on your own GPU), the tempting move is to swap --model-path and restart. BAAI's AREX-2 is a good test of that habit. It is a 27B model trained to work on a problem in rounds (propose, measure, reflect, revise), released on 29 September under Apache 2.0, not gated, and its config.json opens with Qwen3_5ForConditionalGeneration, the same class as Qwen3.8-27B from the Qwen team.

We had been serving base Qwen3.8-27B with sglang, an open-source inference server, so we diffed the two configs. One caveat first: we have not run AREX-2. Every speed below is from the base model. Tables, command and sources are in the full AREX-2 serving analysis on inferencekey.com.

The diff that says "drop-in"

BAAI shipped a clean release: layer layout, head counts, context length, vocab and vision tower all match Qwen3.8-27B. The config differs only in transformers_version and two redundant top-level keys, and chat_template.jinja, the file that formats the conversation and tool calls, is byte-identical.

Stop reading there and you ship the old command.

Check the weights, not the config

The base model carries a multi-token prediction (MTP) head, a small extra layer that guesses a few tokens ahead so the server can emit several per step. AREX-2's config still declares it. Its weight files do not contain it.

config.json                    "mtp_num_hidden_layers": 1
safetensors index (AREX-2)     0 mtp tensors
safetensors index (Qwen3.8)    15 mtp tensors

The safetensors index lists every weight tensor in the release, so it is the honest inventory. A config field is a promise; check that something actually backs it.

The second quiet change is in tokenizer_config.json, which controls how text is cut into tokens. A community repack reports that the pre-tokenizer and decoder settings differ from Qwen3.8. Likely harmless. Also what surfaces as a garbled tool call hours into an agent run.

Flags that were right for the base model

Our quickest base-model setup ran on an RTX PRO 5000 in a CUDA 13 pod, with RadixArk's NVFP4 quant (weights compressed to 4-bit floats) and the z-lab DFlash2 drafter. It decoded (wrote output) at 115 to 136 tok/s. Little of that survives the swap:

• No NVFP4 build of AREX-2 exists yet. Today you pick from BAAI's BF16 original (54.7 GB), prithivMLmods' FP8 (35.9 GB), cyankiwi's AWQ INT4 (20.2 GB) or numsu's INT4 W4A16 (18.6 GB).
• The drafter was trained against another model. Speculative decoding has a small drafter propose tokens that the big model then verifies in one pass; it only pays off when most guesses are accepted. z-lab's DFlash2 drafter targets base Qwen3.8-27B, and a fine-tune changes which tokens the model favors, so nobody knows its hit rate on AREX-2. numsu's INT4 card does publish speeds with a DFlash2 drafter on two RTX 3090s, but without an acceptance length.
• mamba_ssm_dtype. AREX-2 asks for float32. Our base command forced bfloat16, fine on Qwen3.8, untried on a model tuned for long reflective runs.
• Parsers are a guess. These sglang options turn raw model text into separate reasoning and tool-call fields. qwen3 and qwen3_coder ought to work since the template matches. No sglang run of AREX-2 has been published.
• Pick long context or speculation. The base model's 1M context is a YARN override that stretches its native 262,144 positions, and DFlash2 would not run with it. So: 1M without a drafter, or 256K with one.

FP8 on Ampere is the expensive mistake

The RTX A6000 is Ampere and has no FP8 hardware, so sglang keeps the weights in FP8 and unpacks them on the fly through Marlin kernels. Base Qwen3.8-27B decoded there at 24 tok/s. RedHatAI's INT4 W4A16 quant, same card, same 1M context: 41 tok/s. prithivMLmods' AREX-2 FP8 card agrees that full speed needs Ada, Hopper or later. An A100 is Ampere too, so reach for INT4 there.

Weights are not the whole bill. In our base-model INT4 run at 1M context, sglang took about 46 GB, of which only 17.8 GB was weights. Another 11.5 GB held state for the linear attention layers, and 12.8 GB went to the KV cache, the per-token memory the model keeps so it does not reread the whole context on every step. The per-GPU sizing for AREX-2 goes card by card from 80 GB to 24 GB, as estimates.

What to measure on the first run

Start dull: BF16 on an 80 GB card, the native 256K context, no speculative flags, the same parsers, one long agent task. Then change a single thing per run, like a soundcheck that starts with the kick drum. Watch:

• tool calls parsing cleanly late in a long session, given the tokenizer change
• decode tok/s against the base model on the same card, which the shared architecture says should match
• acceptance length if you add a drafter, the number nobody has published
• GPU hours per task, since the benchmark protocol allows up to 1,500 model calls and 24 hours per case

That last item drives the economics. AREX-2's self-reported 70.7 on Frontier-CS and 84.0 on BrowseComp come from that long protocol in BAAI's evaluation repo, with no independent reproduction we could find. Renting a GPU by the hour beats per-token pricing only while the card has work queued, for instance several cases side by side. For the rest of a single-GPU agent setup, see our earlier post on an 80B coding model on one rented A100 driven by Claude Code.

This is a plausible one-card project thanks to BAAI and the Qwen team for the models, prithivMLmods, cyankiwi, numsu, RedHatAI and RadixArk for the quants, z-lab for DFlash2, numsu again for an experimental MTP head trained on AREX-2 outputs, AI Search for flagging the release, and the sglang and vLLM projects. The config tables, the base-model command verbatim, the worker settings for slow warmups and the full limits list are in the complete AREX-2 on one GPU write-up.