跳到正文
DEV Community· SparkLLM·· 5 小時前AI 評分47

在 Rockchip RK3576 和 RK3588 上運行 Spark-X2.5 1.7B

Running Spark-X2.5 1.7B on Rockchip RK3576 and RK3588

AI 導讀

Rockchip 的 RKNN-LLM v1.3.1 基準測試列出 Spark-X2.5 1.7B 在 RK3576 和 RK3588 上的首 token 延遲、生成速度及記憶體用量。

正文

SparkLLM

How long do you wait for the first token, and how fast do the next ones arrive? Rockchip’s RKNN-LLM v1.3.1 benchmark lists both measurements for Spark-X2.5 1.7B.

Board Quantization First token Generation Memory
RK3576 W4A16 768.66 ms 13.57 tokens/s 1,028.58 MB
RK3588 W8A8 386.23 ms 15.61 tokens/s 1,830.32 MB

Rockchip benchmark measurements

Rockchip used 128 input tokens and requested 64 new tokens, with the CPU and NPU running at their respective maximum frequencies. The two rows use different quantization formats. For a board you plan to deploy, start with its listed format and test your own prompt length. Record time to first token separately from generation speed.

Rockchip’s September 29 benchmark table

來源:DEV Community · dev.to