跳到正文
DEV Community· InApp·· 2 天前AI 評分33

為何拍下單張 GPU 每秒生成 776 個 tokens 的 60 秒實測影片

776 tokens a second on one GPU — why I filmed the 60-second proof

AI 導讀

作者以 60 秒螢幕錄影展示單張 GPU 的生成速度,平均每秒 776 個 tokens。測試使用 vLLM、4-bit 量化模型及 continuous batching,透過 /v1/completions 端點進行 30 秒 curl 測試;GPU 全程維持 100% 使用率,耗電約 350W。

正文

Cover image for 776 tokens a second on one GPU — why I filmed the 60-second proof

InApp

I kept seeing claims about single-GPU throughput that sounded too good, so I decided to just show the meter running. The new short is up: https://www.youtube.com/shorts/WoTq58fIag0

The concrete detail: I'm running a 4-bit quantized model through vLLM with continuous batching enabled, and the 776 tok/s figure comes from averaging generation speed over a 30-second curl loop against the /v1/completions endpoint — not a synthetic benchmark, but real streaming output you can watch tick by.

The cost nobody mentions in these threads is that the GPU is pegged at 100% utilization the entire time, pulling roughly 350W from the wall. That one minute of footage cost me about 6 watt-hours and a few cents in electricity, but if you're renting the card by the hour, that minute is a line item. The walkthrough exists because "trust me, it's fast" doesn't scale; a 60-second screen recording does.

來源:DEV Community · dev.to