低位元量化沒有弄壞權重，卻可能令小模型寫程式失靈

I keep a folder called dead-agents. Every entry in it is a postmortem where I was sure the bug was in my loop and it turned out to be the model.

The most recent one cost me a weekend. I pulled an official tag from a public registry, the kind with the little verified badge energy, q4_K_M in the name. It chatted beautifully. Ask it about Python decorators and it explains them better than I would. Ask it to write one and it produced code shaped like Python the way a dream is shaped like your house. Right room count, wrong everything.

I spent six hours rewriting prompts. Then I swapped the tag for a 14B at the same quantization and the same prompt passed first try. The prompt was never the problem.

A paper made the rounds recently that would have saved me the weekend: it swept a pile of official registry tags against a code benchmark and found a handful of them scoring barely above zero. Not "a bit worse than the card." Zero-ish. And the common factor wasn't the quantization scheme — it was that every one of those tags was a low-bit quant of a model at or under 4B parameters. Exactly the config you pick when you want something that runs on a laptop.

So here's the lesson I now build around, and it's the whole article: "Q4 is basically lossless" is not a property of quantization. It's a property of model size. At 70B, four bits costs you noise. At 3B, four bits costs you the thing.

Mini-lesson follows. Fifteen minutes of setup, and you never get ambushed by this again.

Step 1: Know what you actually downloaded

ollama show <tag> gives you parameter count, quantization, context length. Read it. Two things bite here:

Tags move. A publisher repushes under the same tag and your digest changes silently. Run ollama list and pin the ID next to the tag in your config, your compose file, your Dockerfile, wherever. If the ID changes, you want to know about it in CI, not in production at 2am.

The family name tells you nothing about size. some-model:q4_K_M and some-model:70b-q4_K_M look like siblings. They are not. One is a demo, the other is a tool.

Write the digest and the parameter count down in the same place you write down the pass rate from step 3. That file becomes the most honest model card you own.

Step 2: Build a smoke test you'll actually run

Not a benchmark suite. Twenty problems, each with a signature, a short docstring, and two or three asserts. You grade it by counting booleans. No LLM judge — a judge at temperature 0 is a second unreliable system grading the first, and now you have two things to debug.

# smoke.py — deterministic, no judge, 20 cases
import subprocess, sys, textwrap
import ollama

CASES = [
    {
        "prompt": "Write a Python function parse_kv(s) that turns "
                  "'a=1,b=2' into {'a': '1', 'b': '2'}. "
                  "Return only the function.",
        "tests": '''
assert parse_kv("a=1,b=2") == {"a": "1", "b": "2"}
assert parse_kv("") == {}
assert parse_kv("a=1,a=2") == {"a": "2"}
''',
    },
    # ... 19 more, same shape
]

def extract(text):
    if "```

" not in text:
        return text
    block = text.split("

```")[1]
    if block.startswith("python"):
        block = block.split("\n", 1)[1]
    return block

def passes(model, case):
    out = ollama.generate(
        model=model,
        prompt=case["prompt"],
        options={"temperature": 0, "seed": 42, "num_predict": 400},
    )["response"]
    src = extract(out) + "\n" + case["tests"]
    try:
        subprocess.run([sys.executable, "-c", src],
                       timeout=5, check=True, capture_output=True)
        return True
    except Exception:
        return False

score = sum(passes(sys.argv[1], c) for c in CASES)
print(f"{score}/{len(CASES)}")

The timeout=5 is not paranoia. A degraded quant will cheerfully write you a while True that looks like a solution and never returns.

Twenty cases is enough to separate 2/20 from 16/20, and small enough that you'll run it. That last part is the only part that matters.

Step 3: Run the control, because the number alone means nothing

A score with no comparison is a feeling with a decimal point. Run the same script against three things:

• the artifact you actually downloaded,
• the biggest model you can afford at the same quant,
• ideally, the unquantized version of the same base model if it exists.

That's how you tell "this model can't code" apart from "this quant can't code." On my box, a 3B at Q4 scored 3/20. A 14B at the same Q4 scored 16/20. A 7B at Q8 landed around 11/20. Size beat bits, twice, on the same afternoon. I've stopped being surprised by that and started budgeting RAM for it instead.

Step 4: Test the thing your agent actually needs

Code generation is the visible failure. Structured output is the silent one, and it's the one that kills agents.

Take ten prompts, require JSON with specific keys, parse it. Count three things separately: unparseable, parseable but missing keys, and correct. A model that returns {} parses fine and does nothing, which is worse than a crash because your loop keeps going. Do the same with your actual tool-call schema. Ten calls, count the valid ones. If it's not 10/10 on a schema that simple, it's not a planner, whatever the card says.

Step 5: Set a floor, not a vibe

Mine, roughly: under 12/20 on the code smoke test never touches a repository. Under 9/10 on structured output never gets to be an agent's planner. Below both, the model can classify, summarize, route, or draft — and that's a real job. It just isn't the job people buy the laptop for.

Put the numbers in CI. Tags move. Digests don't.

What I do when it fails

Go up in parameters before you go up in bits. Every time I've measured it, a bigger base model at Q4 has beaten a smaller one at Q8 on the same hardware, and it costs about the same in RAM. If the RAM isn't there, then you don't have a code model — you have a chatbot that can also write prose about code, which is fine, but label it that way in your README.

And be honest about the registry problem underneath all this. An official tag means a publisher uploaded it. It doesn't mean anyone ran a single test on the 1.5B quant before it went public. Assume nobody did, yourself included, and let the twenty-case script be the check that nobody else ran.

Mine now runs before the agent starts. Three minutes, no GPU drama, and it has already caught two regressions I would have blamed on my own prompt engineering. Worth it.