Basalt 操作導覽:為何 776 tok/s 需要獨佔整張 GPU
Basalt walkthrough is up: why 776 tok/s needs the whole card to itself
Basalt 的操作導覽解釋其 776 tok/s 表現,以及為何需要獨佔一張 GPU:整個 KV cache 鎖定於單張 GPU 的 VRAM,不會分片至其他 GPU。
I kept getting the same question in DMs after the benchmark posts: "How is Basalt actually pulling 776 tokens a second?" Not the number itself — the mechanism. People wanted to see why it behaves differently from vLLM or llama.cpp on the same hardware, so I recorded a full walkthrough instead of pasting another config snippet.
The detail that surprised me while editing the footage: Basalt doesn't just stream faster, it pins the entire KV cache to a single GPU's VRAM and refuses to shard across cards. That's why "only on one GPU" in the title isn't a footnote — it's the core design. On an RTX 4090 it holds a full 32k context window in roughly 18 GB, which sounds fine until you realize that leaves almost nothing for a second model running concurrently.
The real cost to the operator is exclusivity. You can't co-host anything else on that card while Basalt is serving. If you want full-speed inference plus a separate service, you're either buying another GPU or scheduling Basalt alone. I walk through the memory map, the queueing behavior, and exactly where the bottleneck sits in the video: https://www.youtube.com/watch?v=sMvxrrUq9M0
來源:DEV Community · dev.to