一項設定造成 19 倍延遲差距：我在單 GPU 上評測 vLLM 的心得

I built a small serving harness that measures latency, throughput, AND output drift per config. It found a 19x latency cliff on a single knob, a 3.6x model speed gap — and a reasoning model that can't reproduce its own outputs at temperature 0.

Scope note, first paragraph as promised: one RTX 3060, two models (Swallow-8B AWQ, llm-jp-1.8B bf16), short prompts, synthetic Poisson arrivals — not production traces. No human judged anything; drift is exact output-hash match.

Gentle benchmarks lie

First surprise: max-num-seqs 32 vs 128 showed identical numbers across fixed-concurrency and Poisson loads. The server never broke a sweat, so the knob looked irrelevant. It isn't — my load was just too polite. I added a stress arm (64 concurrent, 256 output tokens) and the same comparison became TTFT p50 8.3s vs 0.43s. Same models, same prompts, one knob: 19x. TPOT stayed flat throughout (16→28ms) — textbook prefill queueing. Lesson I'll reuse everywhere: config comparisons REQUIRE a stress arm; gentle benchmarks can't tell servers apart.

Small models win big

Same stress, different model: llm-jp-1.8B serves at 119ms TTFT vs Swallow-8B's 432ms, 28.5 vs 15.1 req/s. Unsurprising direction, useful magnitude — and a deployment footnote: llm-jp runs under vLLM's own kernels while stock transformers trips on its RoPE config. Know your stack's two paths.

Temperature 0 is not deterministic (for reasoning models)

The quality probe — 50 frozen prompts hashed per config, there to catch quality regressions — caught something else: the same Swallow server, identical requests, 10/50 outputs differ run to run (12/50 across restarts). A non-reasoning model on the same harness: identical ×3. Think-mode sampling leaks through temperature 0. If your production system assumes deterministic serving (caching, evals, replay), verify it per model — don't assume it.

The harness

Asyncio+aiohttp client, fixed + Poisson loads, TTFT/TPOT p50/p95, error rates, per-run JSONL, frozen quality set. Client-only venv; the server lives in sibling venvs. 3 tests against a fake SSE server run in CI (no GPU needed). All numbers, prompts, and hashes are linked below.