跳到正文
DEV Community· InApp·· 2 天前AI 評分50

Basalt 操作導覽:為何 776 tok/s 需要獨佔整張 GPU

Basalt walkthrough is up: why 776 tok/s needs the whole card to itself

AI 導讀

Basalt 的操作導覽解釋其 776 tok/s 表現,以及為何需要獨佔一張 GPU:整個 KV cache 鎖定於單張 GPU 的 VRAM,不會分片至其他 GPU。

正文

Cover image for Basalt walkthrough is up: why 776 tok/s needs the whole card to itself

InApp

I kept getting the same question in DMs after the benchmark posts: "How is Basalt actually pulling 776 tokens a second?" Not the number itself — the mechanism. People wanted to see why it behaves differently from vLLM or llama.cpp on the same hardware, so I recorded a full walkthrough instead of pasting another config snippet.

The detail that surprised me while editing the footage: Basalt doesn't just stream faster, it pins the entire KV cache to a single GPU's VRAM and refuses to shard across cards. That's why "only on one GPU" in the title isn't a footnote — it's the core design. On an RTX 4090 it holds a full 32k context window in roughly 18 GB, which sounds fine until you realize that leaves almost nothing for a second model running concurrently.

The real cost to the operator is exclusivity. You can't co-host anything else on that card while Basalt is serving. If you want full-speed inference plus a separate service, you're either buying another GPU or scheduling Basalt alone. I walk through the memory map, the queueing behavior, and exactly where the bottleneck sits in the video: https://www.youtube.com/watch?v=sMvxrrUq9M0

來源:DEV Community · dev.to