VIDRAFT POCKET 的 POCKET-Darwin-180B 可在 8 GB VRAM、32 GB RAM 手提電腦運行，系列下載量逾 3.15M

VIDRAFT POCKET: Running a 180B MoE Model on an 8 GB VRAM Laptop — 3.15M Downloads and Counting

TL;DR: VIDRAFT POCKET is a family of GGUF-format sparse Mixture-of-Experts models designed to run on consumer hardware — including CPU-only PCs and laptops with as little as 8 GB VRAM — without any modified runtimes or cloud dependency. The flagship POCKET-Darwin-180B achieves 87.65% quality retention at 4-bit quantization, matching its BF16 baseline. With over 3.15 million total Hugging Face downloads, it's worth understanding how it actually works.

What it is

VIDRAFT POCKET is a collection of GGUF-formatted large language models built around three explicit design constraints:

• Standard-runtime-only: Every file loads into the official, unmodified llama.cpp build — and by extension, Ollama, LM Studio, and PocketPal. No custom forks, no CUDA requirement, no cloud calls.
• Sparse Mixture-of-Experts (MoE) architecture: Only a small fraction of the total network parameters are activated per token, making very large parameter counts (35B, 180B) viable on hardware that would otherwise be completely inadequate.
• Measured, not estimated, performance: Benchmark numbers are sourced from same-machine, same-binary comparisons rather than theoretical throughput projections.

The family currently includes at least three major repositories on Hugging Face. As of November 1, 2026, those repositories logged a combined 1,113,164 downloads in the prior 30 days and 3,154,913 total downloads. POCKET-Darwin-180B had been live for fewer than 30 days at that point, so its 30-day and lifetime download figures are identical.

How it works

The core engineering insight behind POCKET is that sparse MoE models have a favorable ratio of total parameters to active parameters per forward pass. A 180B-parameter MoE model might activate only a fraction of those weights for any given token, which dramatically reduces the memory bandwidth and compute needed at inference time compared to a dense model of equivalent parameter count.

POCKET packages these models as GGUF files, the standard interchange format for llama.cpp-based runtimes. GGUF supports mixed-precision quantization, allowing different layers or weight groups to be quantized at different bit widths. This is how POCKET-26B achieves near-baseline quality at 2-bit quantization, and how POCKET-Darwin-180B's 4-bit variant matches its BF16 counterpart on quality benchmarks.

One notable operational detail surfaced in the article: when running inference on CPU with llama-cli, the -t flag should be set to physical core count, not logical thread count. Using all hyperthreaded logical cores can significantly decrease throughput rather than increase it — a common footgun when tuning llama.cpp for multi-core CPUs.

Benchmarks & results

All figures below come directly from the source article and reflect same-binary, same-machine comparisons:

Note: On H100 prompt processing (prefill), the comparison model was faster — 1,816 tok/s vs. 753 tok/s for POCKET-35B. Generation throughput and prefill throughput are distinct metrics; check both for your use case.

Quantization quality:

• POCKET-26B at 2-bit with mixed-precision quantization: near-baseline quality score retention
• POCKET-Darwin-180B at 4-bit: 87.65% on the quality benchmark, matching the BF16 baseline

File sizes and minimum hardware:

• POCKET-35B Q2_K: 13 GB — runs on a CPU-only PC with standard llama.cpp; measured at 19.5 tok/s on a MacBook M3 Pro CPU
• POCKET-Darwin-180B 4-bit: 111 GB — measured at 4.17 tok/s on a laptop with 8 GB VRAM + 32 GB RAM; also runnable fully on CPU in a mini-PC with 128 GB RAM

How to try it

POCKET models are publicly available on Hugging Face. Files are standard GGUF and load into any official llama.cpp-based runtime without modification.

Download via Hugging Face CLI:

pip install huggingface_hub
huggingface-cli download <VIDRAFT-POCKET-repo-name> --local-dir ./pocket-model

(Check the official VIDRAFT Hugging Face organization page for exact repository names and recommended quantization variants for your hardware.)

Run with llama.cpp (CPU-only inference):

./llama-cli -m ./pocket-model/<file>.gguf -ngl 0 -t <physical_core_count>

• -ngl 0 forces all layers to CPU (no GPU offload)
• -t should equal your physical core count, not your logical thread count

The same GGUF files are compatible with Ollama, LM Studio, and PocketPal without any extra configuration. For users who prefer not to install anything locally, the article mentions that a POCKET Image Studio option is available for browser-based access within the POCKET collection.

FAQ

Q: Do I need a modified or custom-patched version of llama.cpp?
A: No. The source article explicitly states that the official, unmodified llama.cpp binary loads all POCKET files. This also means compatibility with Ollama, LM Studio, and PocketPal out of the box.

Q: Why does setting -t to logical thread count hurt performance?
A: Hyperthreading shares physical execution resources between logical cores. For memory-bandwidth-bound workloads like LLM inference, saturating physical cores and then adding logical threads creates contention rather than parallelism. Set -t to physical core count for best results.

Q: Is 4.17 tok/s on a 180B model actually usable?
A: That depends on your workload. For interactive chat, 4–5 tok/s is slower than comfortable reading speed but functional. For batch processing or document summarization tasks where you aren't watching tokens stream in real time, it's more practical than it sounds for a 180B model on consumer hardware.

Q: What quality benchmark does the 87.65% figure refer to?
A: The source article does not name the specific benchmark suite. Treat it as an internal quality consistency metric relative to the BF16 baseline, and run your own domain-specific evals before committing to production use.

Originally reported by 넷이즈 网易 (중국) (2026-11-01) — source article.