跳到正文
DEV Community· Jacek Garbuliński·· 37 分鐘前AI 評分59

多個 LLM 共用 GPU 時,vLLM 睡眠模式的代價

What vLLM sleep mode costs when multiple LLMs share a GPU

AI 導讀

作者在與另外三項服務共用的 H200 NVL 上,實測 vLLM 對 Qwen3.8-27B 的睡眠與喚醒成本。睡眠需 2.3 至 2.4 秒並釋放 46.8 GiB,喚醒 28 GiB 權重需 1.28 秒,但會清空 KV cache。文章建議讓長文件模型保持載入,偶爾使用的模型共用一個模型槽位;131,000 tokens 的提示詞在喚醒後重讀需 17 至 18 秒。

正文

Every language model loaded on a graphics card holds part of the card's memory, whether anyone is calling it or not. vLLM, an open-source model server, can put such a model to sleep and wake it when it is needed. On our card, a 27-billion-parameter model woke in 1.3 seconds, where starting it from scratch took 82 to 123 seconds. A sleep does not bring back what the model had read, so its first answer about a long document waits 17 to 18 seconds again. For a team serving multiple LLMs from one GPU, that trade decides which models can sleep and how much memory the others gain.

This guide measures what vLLM sleep mode saves and costs for Qwen3.8-27B on one H200 NVL card, which it shares with three other services in Blixild AI, our LLM API. The take-aways, with the numbers behind them:

Take-away Qwen3.8-27B FP8 on one H200 NVL
A wake is cheap: 1.3 seconds, not minutes 28 GiB of weights back on the card in 1.28 s; a sleep 2.3 to 2.4 s; a start from scratch 82 to 123 s
A sleeping model gives back almost all of its memory 46.8 GiB freed, 2.4 GiB left on the card
The server's RAM holds about one and a half times the weights 43.2 GiB for 28 GiB of weights, from the first sleep on, awake or asleep
What a sleep costs is the emptied KV cache, and the cost grows with the prompt a cached 19,800-token prompt: 1.46 s after a wake, against 0.25 s with the cache kept; 17 to 18 s to read 131,000 tokens again
Keeping the cache in the server's RAM would remove most of that cost worked out: 131,000 tokens back on the card in about 0.2 s
So models for long documents stay loaded, and occasional ones share one slot worked out: Qwen's cache 494,000 tokens with every model loaded, about 1.1 million with the slot

We timed five sleeps and wakes each of Qwen3.8-27B and Arete 27B from the card's own machine, so the network adds no time to them. Five cycles of two models show the size of the costs, not exact values; firm numbers would take more cycles, models and cards, and wakes under load. The read times come from earlier measurements on the same card, and the cache copied from RAM and the slot layout are worked out.

It is written for teams that serve several open models with vLLM from one GPU and run short of memory, and for teams that swap models by restarting the server. Our Qwen3.8-27B long-document guide shows how a long document is read and kept between questions. The sections cover, in order: why each model's memory is fixed at start, what a sleep and a wake move, what a sleep loses, a layout that keeps what matters, and where to start.

Each model on a shared GPU fixes its KV cache at start

A model on a GPU needs memory for its weights, the numbers it computes with, and for its KV cache, which holds what it has read in each conversation. vLLM sizes the cache once, when the server starts, from --gpu-memory-utilization, the share of the card one server may take, or from a cache size in bytes. The cache keeps that size whether the model is busy or idle.

Our H200 NVL holds four services. Qwen3.8-27B, the one meant for long documents, had a cache of 2.0 million tokens while it shared the card with smaller services. With two more models on the card, one of them holding 131,000 tokens, Qwen's cache is 494,000 tokens.

The cache size sets how much conversation fits on the card at once, not how fast one request runs. The memory held by models nobody is calling is what sleep mode can give back.

vLLM sleep mode copies the weights to RAM and back

At level 1, vLLM sleep mode copies the model's weights to the server's own memory, its RAM, and discards the KV cache; a wake copies the weights back and sets the cache up again, empty, at its old size. Level 2 also discards the weights, for training that loads new ones. Its sleep and wake calls are development endpoints, opened by the variable VLLM_SERVER_DEV_MODE=1, which vLLM says should not be exposed to users. vLLM wakes nothing by itself: in our trial, a request sent to a sleeping model waited and was answered right after the wake.

The link between the card and the server's processor sets how long a wake takes. Qwen3.8-27B's 28 GiB of weights came back in 1.28 s and Arete's 16.8 GiB in 0.75 s: 24 GB a second, three quarters of the 32 GB a second each way of a PCIe Gen4 link with 16 lanes, half NVIDIA's 64 GB/s for both ways together. Our H200 NVL supports PCIe Gen5 but runs at Gen4, the most its server supports, as nvidia-smi, NVIDIA's command-line tool for the card, shows.

Qwen's sleep took 2.3 to 2.4 s and freed 46.8 GiB, leaving 2.4 GiB on the card. The first sleep after a start took 26.9 s, because the server sets up its RAM for the copy then and keeps it: about one and a half times the weights, 43.2 GiB for Qwen, awake or asleep. We timed each wake with no other request to that model. The weights come back in 1.3 s, and the KV cache does not come back at all.

The cost of a sleep is the emptied KV cache

Because level 1 discards the KV cache, the first request after a wake reads its whole prompt again, while new prompts cost what they always did. A 19,800-token prompt that Qwen had cached before the sleep took 1.46 s after the wake, against 0.25 s with the cache kept. A 131,000-token document takes 17 to 18 s to read again and a 254,000-token one 51 to 55 s, against under 1 s and 1.5 s with the cache kept, once for every conversation the model held when it slept.

Two bar charts on one scale for Qwen3.8-27B on an H200 NVL after a wake: with the prompt read again, 1.6 s at 4,000 tokens, 18.6 s at 131,000 and 54 s at 254,000; with the cache copied back from RAM, 1.5 s, 2.4 s and 3.2 s
Seconds from a wake to the first token of the next answer, for one request alone on the card: the measured wake, 1.28 s, plus either the prompt read again (measured, the middle of each range) or the cache copied back from RAM at the measured 24 GB a second and the first token read from it (the copy worked out, the first token measured).

With other requests running, each read takes longer. The cost falls on the models whose users send long prompts, so those models decide the layout.

Keep long-context models loaded and give the rest one shared slot

Three rules follow from these numbers:

  • Keep loaded the models whose cache is the product. That means models for long documents and long sessions, and any service whose server has no sleep mode.
  • Give the occasional models one shared slot. One of them is awake at a time and gets the whole slot for its cache, and a router in front of vLLM switches them when a request needs the other.
  • Keep a sleeping model's cache in the server's RAM. vLLM can offload KV cache blocks to the processor's memory, and recent releases hold them across a sleep when the sleep call passes reset_connector=false, so the first request after a wake loads them instead of reading its prompt again. Qwen's 131,000-token conversation takes about 5 GB, about 0.2 s at the copy speed we measured, against 17 to 18 s. A service that promises not to keep customers' data keeps that copy in RAM only, never on disk, and each customer's cache apart.

On our H200 NVL, a slot as large as Bielik's share today, 41.6 GiB, would change the caches as follows:

One H200 NVL, 140 GiB Every model loaded Qwen loaded, one 41.6 GiB slot
Qwen3.8-27B, long documents 494,000 tokens about 1.1 million tokens
Arete 27B, when awake 114,000 tokens about 700,000 tokens
Bielik 11B v3.1, a Polish model, when awake 131,000 tokens 131,000 tokens
basal, which has no sleep mode 23 GiB 23 GiB

The slot stops paying when two of its models both get steady traffic, because every request to the sleeping one then costs a wake and an emptied cache. This layout is arithmetic on today's memory readings: we have not tried vLLM's offloading with Qwen3.8's mix of attention and linear-attention layers, and what a sleeping Bielik keeps on the card is estimated.

Where to start with vLLM sleep mode on your own GPU

Sleep mode pays for models called now and then with short prompts: they give back their memory and return in 0.75 to 1.3 seconds. A model that answers long documents, or is busy most of the time, stays loaded, because each sleep makes it read its prompts again. Teams with several models on one GPU take all six steps; teams that restart the server to swap models take steps 2, 5 and 6, with a sleep and a wake in place of a stop and a start.

  1. List your models: each with its weights in GB, how often it is called and the longest prompts it gets.
  2. Work out each wake: its weights divided by about three quarters of your link's top speed, 24 GB a second on PCIe Gen4 with 16 lanes.
  3. Work out each re-read: how long the model takes to read those prompts on your card, from your server's timings or with the LLM GPU calculator.
  4. Sort them: keep loaded the models whose prompts take many times their wake to read, or that are busy most of the time, and put the rest in one slot sized for the largest.
  5. Start each model with sleep mode on: --enable-sleep-mode and VLLM_SERVER_DEV_MODE=1, once the server's RAM holds about one and a half times the weights of every model that sleeps; start the slot's models one at a time and put each to sleep before the next starts, so the slow first sleep is over before anyone waits.
  6. Put a router in front: only it calls sleep and wake; for a sleeping model it puts the awake one to sleep with mode=wait, so its requests finish (the default cuts them), then wakes the one asked for.

It has worked when a wake takes about what step 2 predicted and the first request after it about what step 3 adds, and when the cache size vLLM logs at start for each loaded model has grown as the layout said. We work out the same layout for a customer's own GPU as part of our AI workload hosting.

Sources

  1. vLLM (2026). Sleep mode. https://docs.vllm.ai/en/stable/features/sleep_mode/
  2. vLLM (2026). Engine arguments. https://docs.vllm.ai/en/stable/configuration/engine_args.html
  3. NVIDIA (2026). A100 and H200 product pages, Specifications. https://www.nvidia.com/en-us/data-center/a100/ and https://www.nvidia.com/en-us/data-center/h200/

來源:DEV Community · dev.to