如何用小型 NVIDIA GPU 在 Windows 本機執行千問（Qwen3-4B）

Most guides to running a language model locally assume a big GPU or a long setup: Python, the CUDA Toolkit, a build step. This one does not. In three steps you will have Qwen3-4B, a 4-billion-parameter chat model, answering in your browser from a 1.24 GB file, on the kind of NVIDIA GPU that ships in an ordinary gaming laptop.

We use our own open-source build for this (Pingala-8, a patched llama.cpp), so we will also be clear about where it falls short.

Who this is for

You have a Windows 10 or 11 laptop or desktop.
It has an NVIDIA GeForce RTX 30, 40 or 50 series GPU.
You want a chat model that runs on your own machine, with no account, no cloud and nothing to compile.

What you need

A recent NVIDIA driver. That is the only thing to install. The zip already contains the CUDA runtime files (cudart64_12.dll, cublas64_12.dll, cublasLt64_12.dll), so you do not need the CUDA Toolkit.
GPU memory (VRAM). The Qwen3-4B model file is 1.24 GB; the Llama-3.1-8B file is 2.39 GB. The GPU also needs room for the chat context; see the tips below.
Disk space. The zip is 1.59 GB and unzips to 1.89 GB. The model downloads into the same folder: 1.24 GB for Qwen3-4B, 2.39 GB for Llama-3.1-8B. Plan for about 5 GB free for Qwen3-4B; you can delete the zip once it is unzipped.
Internet, once. The first run downloads the model from Hugging Face. Later runs start from the files on disk.

Step 1: Download the build

Open the v2.4-oneclick release page and, under Assets, download pingala8-windows-cuda12.zip (1.59 GB).

Step 2: Unzip and double-click the .bat

Right-click the zip, choose Extract All, and open the pingala8-windows-cuda12 folder. Inside are llama-server.exe and three other llama.cpp tools, the three CUDA DLLs, a README.txt, license files and three small scripts:

run_qwen3_4b.bat starts Qwen3-4B (about 1.2 GB download).
run_llama31_8b.bat starts Llama-3.1-8B (about 2.4 GB download).
env.bat holds settings that both scripts load; you do not run it yourself.

Double-click run_qwen3_4b.bat. A black console window opens. Here is what the script does, in order (it is 15 lines of plain text; open it in Notepad to check):

It moves into its own folder and creates a models folder there if there is none.
It downloads the two model parts, 4B-csq-vr3-00001-of-00002.gguf and 4B-csq-vr3-00002-of-00002.gguf, from the Genrius-AI/Qwen3-4B-Pingala8-2bit repository on Hugging Face, using the curl.exe built into Windows 10 and 11. A part that is already in models is skipped, which is why later starts are quick.
It runs env.bat, which sets eight GGML_... environment variables: the settings of our GPU kernels, the same values as in the paper's runs.
It opens http://127.0.0.1:8080 in your default browser.
It starts the server:

llama-server.exe -m models\4B-csq-vr3-00001-of-00002.gguf -ngl 99 -c 4096 --port 8080

-ngl 99 puts all of the model's layers on the GPU, -c 4096 sets a context of 4,096 tokens, and --port 8080 is the port the browser tab points at. Only the first part is passed; llama.cpp finds the second one by itself.

Windows may show a security prompt the first time, because the file came from the internet.

Step 3: Chat

The browser tab shows llama.cpp's built-in chat page, served by llama-server.exe on your own PC (127.0.0.1 means "this computer"). The script opens the browser a moment before the model has loaded, so if the page does not come up, wait a few seconds and refresh. Type a message and the reply streams in. On our test machine, an RTX 5070 Ti Laptop GPU, Qwen3-4B writes about 200 tokens per second.

The demo replays text and timing from a real run. Its chat window is recreated, so the llama.cpp page you get looks different.

By default llama-server listens only on 127.0.0.1, so other devices on your network cannot reach it. To stop, close the black window. To start again, double-click the same .bat; nothing is downloaded the second time.

Tips

4 GB of VRAM: use run_qwen3_4b.bat. The 1.24 GB file leaves room for the context and for what Windows itself is using.
8 GB of VRAM or more: both models fit, so you can also try run_llama31_8b.bat (2.39 GB, three files, in the same models folder).
These picks come from the file sizes. We measured on one laptop GPU and have not tried every card.
Close other GPU apps first. Games, video editors and busy browsers take VRAM. Task Manager, under Performance > GPU, shows "Dedicated GPU memory" in use.
Laptops: plug in. On battery, most laptop GPUs run at lower power and generate fewer tokens per second.

Troubleshooting

"Download failed. Check your internet connection and run this file again." The script prints this and waits for a key when a download fails. Fix the connection and double-click the .bat again.
The black window closes right away. The server stopped with an error, and the window closed with it. Type cmd in the folder's address bar to open a Command Prompt there, then type run_qwen3_4b.bat and press Enter: the window then stays open and you can read the message.
The model will not load after an interrupted download. The script skips any model file that already exists, even a half-finished one. Delete that model's .gguf files in the models folder and run the .bat again.
Errors about CUDA or the GPU. The build needs an NVIDIA RTX 30, 40 or 50 series GPU and a recent driver. Update the driver and try again.
The page cannot connect. Wait for the model to finish loading and refresh. If another program already uses port 8080, the server cannot start; close that program first.

Honest limits

NVIDIA and CUDA only. No AMD, Intel or Apple GPUs.
Long prompts are slower. Reading the prompt runs at 0.74x the speed of llama.cpp's IQ2_XXS format at the same file size, so a long pasted document means a longer wait before the first word. The script's 4,096-token context also limits how much fits in one chat.
One user at a time. Serving several chats at once is much slower; this is built for one person on one GPU.
Not in Ollama or LM Studio yet. The files need our patched llama.cpp. Stock llama.cpp, Ollama and LM Studio cannot load them, which is why the zip ships its own llama-server.exe.
Smaller is not free. On a five-task quiz, the 1.24 GB Qwen3-4B keeps 90% of the 8-bit model's score, so it will get more things wrong than the full-size model. We have not evaluated the safety behaviour of the compressed models.
One test GPU. Our speed numbers are from one RTX 5070 Ti Laptop GPU at batch 1, where it generates text 39% faster than llama.cpp's Q4_K_M at half its size. Your card will differ.

How it works, in one paragraph

The original Qwen3-4B is an 8.05 GB file of 16-bit weights; this one stores about 2 bits per weight, with no retraining. Before rounding, the weights are rotated so that no single value dominates; rounding errors are carried onto weights not yet rounded, so they tend to cancel; and the 2-bit codes follow an 8-state trellis, a small memory from one group of weights to the next that lets the decoder tell apart more values than the bits alone could address. A patched CUDA kernel decodes the codes inside each matrix multiply as the model runs. Our first article walks through each step with small worked examples: How We Shrank an AI Model by 85% and Kept 90% of Its Smarts.