Jaull 開源工具推薦電腦可運行的本地大語言模型
A tool that tells you which local LLM your PC can actually run
Jaull 是一款開源 Python 終端介面工具，會根據電腦硬件推薦可在本機運行的大語言模型方案。它只根據 Hugging Face 中繼資料搜尋，最多提供五個方案，列出具體檔案、執行環境、後端及 GPU offload，並分開呈現記憶體估算與實測結果。
For my final degree project I wanted to know what hardware a small company would need to run language models locally. My method was to download a model, watch it crash or crawl, and try another quantization. So I built Jaull to do that work properly.
Jaull is an open-source Python tool with a terminal UI. It detects your CPU, RAM, GPU and backends, asks a few plain questions (what you'll use it for, what matters most, languages, commercial use) and searches Hugging Face using metadata only. You get up to five concrete plans, each with an exact file, runtime, backend and GPU offload, and an explanation of its position.
Estimates and measurements stay separate
Jaull estimates memory by component (weights, KV cache, overhead, margins) and treats RAM and VRAM as two pools, so it can tell you which one you'll run out of. If you want to check, you can run, benchmark or score any plan on your machine. Each result is saved with the file's SHA-256, the machine and the protocol, and only results that match a plan can reorder recommendations. A benchmark of a Q5_K_M file says nothing about the Q6_K next to it.
Some numbers from my RTX 2060:
| Model (Q5_K_M) | IFEval | Generation speed |
|---|---|---|
| Qwen2.5-0.5B | 25.9% | 246 tok/s |
| LFM2.5-1.2B | 80.2% | 231 tok/s |
| Qwen2.5-1.5B | 41.0% | 151 tok/s |
The 1.2B model beat the 1.5B one by a wide margin on instruction following, which is a good reason not to rank by parameter count.
Things I got wrong
My first offload policy left most of the GPU unused. On Qwen2.5-7B it gave 21.5 tok/s, and putting every layer on the GPU gave 59.4 for 133 MB more VRAM. After fixing the formula, the same run reached 44.9.
I also built a tokens/second predictor. A simple version estimated 170 tok/s for a model that measured 18. I removed predictions from the UI, and Jaull now shows a speed only after measuring it.
Some numbers can't be measured at all. On a GeForce that drives your display, Windows won't report VRAM per process, so Jaull says "not comparable" there.
Try it
git clone https://github.com/Ton-Llop/Jaull.git
cd Jaull
uv sync --locked
uv run jaull
You don't need llama.cpp to get recommendations, only to run plans. I haven't tested Intel (Vulkan) or AMD support on real hardware yet. If you try it, I'd like to hear what it recommends for your machine.
Jaull is not a finished product.
來源：DEV Community · dev.to