本地 LLM 執行前的兩個問題：模型放得下嗎，實際速度有多快？

I keep seeing the same two failures when people try to run a model locally. The first is buying or allocating hardware that cannot hold the model plus its context. The second is trusting a single tokens/s number that was measured under completely different conditions.

This is not a product pitch. It is the checklist I use now, plus a small tool I built because I kept redoing the arithmetic.

1. Separate memory fit from speed

These are two different questions and they deserve two different kinds of evidence.

Memory fit is arithmetic you can do yourself. Start with the stored size of the quantized weights, add the KV cache for the context you actually want, then leave headroom for the runtime, the driver and the desktop. A 70B model at Q4 is roughly 40 GB of weights before you add anything else, so "it runs on a 24 GB card" is usually a configuration that offloads to system RAM and then runs slowly.

Speed is a measurement, and it only means something with its conditions attached: exact hardware, exact checkpoint, quantization, runtime and version, context and prompt length, concurrency, and how the number was produced (prefill / TTFT / decode). A decode number from a single stream is not a throughput number from eight concurrent requests. A vendor claim is not a measurement. If the source does not name all of that, the honest answer is "unknown", not a green check.

Note especially: prefill time and tokens/s are not interchangeable. If you only have TTFT, you are looking at a derived prefill estimate, and it should be labelled that way.

2. Pick the runtime before you shop

The runtime decides what is even possible. CUDA, ROCm, Metal, Vulkan, CPU and NPU are separate lanes with separate floors, and a feature that exists on one lane is not proof for another. On Apple silicon, for example, MLX is the native path; on an NVIDIA server, vLLM or SGLang are the throughput paths; on a mixed CPU+GPU box, an offloading runtime for mixture-of-experts models is a different trade again.

3. Serving Apple silicon as an OpenAI-compatible endpoint

A detail worth knowing if you run MLX: mlx-lm ships mlx_lm.server, an OpenAI-compatible HTTP server. It exposes /v1/chat/completions, /v1/completions and /v1/models, defaults to 127.0.0.1:8080, and takes --model plus --host / --port. It is convenient for pointing an existing OpenAI client at your Mac.

The important caveat is upstream's own: mlx_lm.server implements only basic security checks and is not recommended for production. Keep it bound to localhost unless you add authentication and network controls in front of it. For a single-user local setup that is fine; for anything exposed, it is not.

4. Write the conditions down

Whatever you measure, record the conditions next to the number. Six weeks later, "42 tok/s" without "M5 Max, Q4, 8k context, single stream, mlx-lm x.y" is useless to you and misleading to anyone else.

If you want a starting point, I keep a sourced reference that separates memory fit, practical context, measured speed, vendor claims and unknowns, and links every number to its source: LocalForge. It is a decision guide, not a benchmark lab - it publishes no numbers of its own.