跳到正文
DEV Community· sheephess9527·· 10 天前AI 評分57

24GB 顯示卡上的本地編碼大語言模型比較:Qwen2.5-Coder 32B 與 Qwen3-Coder 30B-A3B,關鍵在上下文

Local coding LLM on a 24GB GPU: Qwen2.5-Coder 32B vs Qwen3-Coder 30B-A3B (it's all about context)

AI 導讀

sheephess9527 比較 Qwen2.5-Coder 32B 與 Qwen3-Coder 30B-A3B 在 24GB 顯示卡上的量化佔用,重點是兩者可容納的上下文長度。

正文

sheephess9527

If you want a local coding model on an RTX 4090 / 3090 / 7900 XTX, the question isn't only "does it load", it's "does it load with enough context to read my files".

Qwen2.5-Coder 32B (dense)

Build 4K 8K 16K 32K
GGUF Q4_K_M 21.7 GB (90%, tight) 95% doesn't fit doesn't fit
AWQ INT4 18.1 GB (75%) 80% 89%, tight doesn't fit
EXL2 3.5bpw 16.0 GB

So the dense 32B is great for chat-length prompts. AWQ via vLLM gives you comfortable 8K (Linux / WSL2 only).

Qwen3-Coder 30B-A3B (MoE)

Build 4K 32K
GGUF Q4_K_M 19.8 GB (83%) 22.7 GB (95%, tight)
AWQ INT4 16.4 GB 19.2 GB (80%, comfortable)

Only ~3B active parameters per token, so its bandwidth ceiling is far higher than a dense 32B. Note: the site has no measured run of it and runs no coding benchmarks, so this is about fit, not code quality.

Want even longer context?

North Mini Code 1.0 30B-A3B GGUF Q4_K_M: ~20.8 GB at 32K, ~23.5 GB at 128K (tight). Only 13 of its 49 layers keep a growing cache.

Quick start

# dense 32B, default 4K window
ollama pull qwen2.5-coder:32b
# AWQ, 8K comfortable
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct-AWQ --max-model-len 8192 --host 127.0.0.1 --port 8000

TL;DR

  • Short prompts, best dense coder: Qwen2.5-Coder 32B AWQ.
  • Whole-file / repo context on one card: an A3B MoE coder.

Full guide with GPU checks and troubleshooting: https://quantized.uk/cookbook/qwen-coder-32b-single-4090/
Model page with all quants: https://quantized.uk/quant-hub/qwen2.5-coder-32b/

來源:DEV Community · dev.to