跳到正文
DEV Community· Alan West·· 1 天前AI 評分68

Cloudflare clairic-flash 如何在本機執行,以及 RTX 3090/4090 的限制

How to Run Cloudflare Clairic-Flash Locally (Its Runtime Fails on RTX 3090/4090)

AI 導讀

文章説明如何在本機執行 Cloudflare 的 9B 開放權重視覺模型 clairic-flash,並指出官方 runtime 使用 DeepGEMM,RTX 3090/4090 因 compute capability 低於 9.0 而無法運行。

正文

If you're planning to run Cloudflare's new open-weight vision model, clairic-flash, locally on a 24 GB RTX 3090, two things will bite you. Cloudflare's official runtime doesn't work on that card, even in BF16. And the GGUFs you'll find for llama.cpp quietly drop the part that makes Clairic worth using.

Hugging Face lists clairic-flash as an image-text-to-text model, so the obvious move is to load it into a chat UI. Don't. It's a 9B extraction engine: you hand it a document plus a typed schema, and it hands back values, confidence scores and null probabilities. Most of that behaviour lives in a clairic_runtime/ folder of Python code and a separate typed head, neither of which any GGUF contains.

I went through the VRAM numbers for BF16 and int4, the compute-capability check that breaks RTX 30- and 40-series cards, the MLX route on Apple Silicon, and what llama.cpp actually gives you.

What Cloudflare released: specs, license and Workers AI pricing

On November 18, 2026 Cloudflare published two open-weight models from a group it now calls the "Workers AI superintelligence team". The post opens with "Cloudflare has become a neolab." I'll leave that one alone.

The facts from the model card and config.json:

clairic-flash clairic
Size 9B 27B
Base model Qwen/Qwen3.5-9B Qwen/Qwen3.8-27B
License Apache-2.0 Apache-2.0
Architecture Qwen3_5ForConditionalGeneration, 32 layers (24 linear-attention Gated DeltaNet + 8 full attention)
Config context 262,144 positions
Runtime default max_length 65,273 tokens
BF16 weights on disk ~19.3 GB in 4 shards, plus a 261 MB clair_head.safetensors
Workers AI ID @cf/cloudflare/clairic-flash @cf/cloudflare/clairic
Workers AI price $0.10 per M input tokens $0.24 per M input tokens ($0.10 until Dec 1, 2026)

The card itself calls the 9B "9B"; the Hugging Face size widget rounds it to 10B params (and the big one to 28B).

Ignore the 262K context number. The backbone supports it, but the official loader load_release_model() defaults to max_length=65_273, and systemone() refuses to change that per call. Your inputs are also capped at four images or PDF pages, 4 MiB per embedded file, 8 MiB combined. Remote URLs are rejected.

Benchmarks: nothing against Qwen-VL or Gemma

If you came for a comparison table against Qwen2.5-VL, Gemma 3 or SmolVLM, there isn't one. The model card has no benchmark section at all.

The blog post has evals, but they're Cloudflare's own sets, compared against Liquid AI's D1.2 (with Apple Vision OCR in front of it) and ScaleDown's Extract. I've left out the Clairic-preview column:

Eval Clairic Clairic-flash D1.2 + OCR ScaleDown
Fresh text fields, 96 requests 335/336 328/336 304/336 259/336
Mixed-input fields, 36 requests 221/226 213/226 116/226 156/226
Tracked customer assertions, 139 request shapes 1,263/1,263 1,263/1,263 973/1,263 653/1,263
Synthetic receipt fields 576/576 576/576 397/576 280/576

Cloudflare's own words: "take these evals as directional guidance since they are self-reported." One row is literally labelled "Internal eval (mchen)". I appreciate the honesty. These numbers still prove very little to anyone outside Cloudflare, and a perfect 576/576 on synthetic receipts mostly tells me the training data had synthetic receipts in it.

Latency is the more interesting claim: on a self-hosted server Cloudflare reports clairic-flash at 155/328 ms p50/p95 on text fields and 305/605 ms on image fields. Note that's the full HTTP response with every field in the request, measured on their hardware, which they don't name.

Why the clairic-flash GGUF quants miss the point

Clairic is a Qwen backbone plus three things:

  1. A request pipeline (clairic_runtime/) that rewrites each field into its own prompt, builds an xgrammar constraint from the field schema, scores null, retries, and assembles lists and tables.
  2. A typed head (clair_head.safetensors) that answers dates, durations, compute, estimate and forecast fields from hidden states in one forward pass.
  3. Grounding code that returns character spans into your source text.

The mlx-community port says it plainly: mlx_lm.load / mlx_vlm.load "will not reproduce Clairic". The Xylolabs GGUF repo is upfront too: the head and runtime "are not in these files."

That doesn't make the GGUFs useless. You get a Qwen3.5-9B that has seen a lot of extraction training, and with llama.cpp's JSON-schema grammar you can approximate the behaviour. But you lose the confidence scores, p_null, spans and the forecast head. Those were the reasons to pick this model over plain Qwen in the first place.

Running clairic-flash the official way (Transformers, CUDA)

There's no vLLM or SGLang recipe in the card. The documented path is Transformers plus Cloudflare's code, which is loaded from the snapshot via sys.path, so read it before you run it.

python3.11 -m venv .venv && source .venv/bin/activate
hf download Cloudflare/clairic-flash --local-dir ./clairic-flash
pip install -r ./clairic-flash/requirements.txt

That requirements.txt pins torch==2.13.0, transformers==5.10.2, xgrammar==0.2.1, flash-linear-attention==0.6.0, sgl-deep-gemm==0.1.5.post3, and a FlashInfer JIT cache built for cu130, so keep it in a fresh venv and away from any environment you care about.

import sys
sys.path.insert(0, "./clairic-flash")
from clairic_model import load_release_model, systemone

model, processor = load_release_model("./clairic-flash", device="cuda")
response = systemone(model, processor, {
    "state": "Invoice from Northwind Supply. Issued 2026-09-02. "
             "Items: Pens; Paper. Total: $1080.00.",
    "fields": {
        "vendor": {"type": "text", "instructions": "The invoice vendor."},
        "total": {"type": "number", "decimals": 2},
        "issued": {"type": "date"},
    },
})
print(response["answers"])

Why the official runtime fails on RTX 3090 and 4090

This is where most home setups will break. According to stuchapin's AutoRound repo, the shipped runtime sends every contiguous BF16 GEMM to DeepGEMM, which raises Unsupported architecture below compute capability 9.0. The repo's own words: the original runtime "fails on RTX 30-series cards even with the BF16 weights." That repo only tested a 3090, but the cutoff is the cutoff: an RTX 4090 is compute capability 8.9, so it sits under the same line. The kernel choices assume Hopper-class data center GPUs.

That repo patches clairic_runtime/math.py to use the runtime's own Triton matmul fallback below 9.0 and quantizes the MLP and full-attention layers to int4. Their reported numbers on one 3090:

BF16 W4A16
Weights in GPU memory 18.2 GiB 11.1 GiB
Peak during test 21.2 GiB 14.2 GiB
13 requests, warm 62.0 s 32.6 s
Fields correct 46/49 47/49

So BF16 needs a 24 GB card and the int4 build fits a 16 GB one. The AutoRound kernels compile on first use: about 11 minutes on their first run, 20 to 135 s per request, cached on disk after that. Their absolute timings are also slower than Cloudflare's because of the fallback kernels, so look at the BF16 vs. int4 ratio, not the seconds. The accuracy misses are the part I'd act on: both precisions got a subtotal + tax + shipping sum and a "net 30" due date wrong, at confidence ≤ 0.18. Extract the inputs with the model and do the arithmetic in your own code.

Clairic-flash on Apple Silicon: MLX works, 4-bit costs accuracy

The mlx-community/clairic-flash-4bit build keeps Cloudflare's pipeline and swaps the engine. It quotes 6 GB at rest and up to 11 GB on their timed requests, so 16 GB is the practical floor. It also ships server.py with a /v1/systemone endpoint.

Against the official runtime on an H100, the 4-bit build matched 247/262 fields. The typical mistakes were misread digits like 2231.5 for 2231.25. On a smaller 77-request subset that leaves out the longest documents, 4-bit matched 231/246 and their 8-bit build (10.8 GB) matched 246/246. A mixed recipe with the first 20 layers at 6-bit and the rest at 4-bit (7.3 GB) got 241/246. If you're extracting invoice numbers, use 8-bit or the mixed 6/4-bit recipe; the extra RAM is cheaper than a wrong digit in an invoice ID.

llama.cpp, GGUF and Ollama

For llama.cpp, Xylolabs ships quants plus a vision projector (mmproj-clairic-flash-BF16.gguf), with Q4_K_M at 5.69 GB and a reported mean KL of 0.0468 against an f32 reference:

llama-server -m clairic-flash-Q4_K_M.gguf --mmproj mmproj-clairic-flash-BF16.gguf \
  --jinja -ngl 99 -c 32768 --chat-template-kwargs '{"enable_thinking":false}'

mradermacher has static quants from Q2_K to Q8_0 plus f16, with no mmproj file in the list, so treat them as text-only. Like the Xylolabs files, they're the backbone without the head. I found no official Ollama listing in the card or blog.

Should you self-host it or just use Workers AI?

The weights are Apache-2.0, so you're free to fine-tune them, ship them and sell products on top. That part is a good release. The rest of the packaging is clearly Workers AI infrastructure with the door left open: Hopper-first kernels, a CUDA 13 wheel pinned in requirements, @cf/cloudflare/clairic-flash accepted as a model ID in the local API, and benchmarks that only compare against other niche extraction products. At $0.10 per million input tokens with output tokens free, Workers AI wins for most people unless the data can't leave your network.

If your data can't leave the building, though, the 16 GB int4 path is real, and it beats prompting a general VLM for JSON and hoping.

Has anyone gotten the official runtime going on a 4090 or an RTX 50-series card, or is everyone outside Hopper stuck on the patched build? And if you've put it through real invoices, I want to hear about the misses.

Sources

來源:DEV Community · dev.to