Qwen3-0.6B 的 Q4_K_M 與 BF16 輸出分歧，作者以 heterodyne 逐層追查

Qwen3-0.6B in BF16 reads this:

def is_prime(n):
    if n < 2:
        return

and puts 97.9% on False. The Q4_K_M quant of the same model puts 79.1% on True.

That's one token out of 1,282 I scored. Across the whole text, Q4_K_M changes the top token at 11.6% of positions, its mean KL from BF16 is 0.067, and perplexity goes from 7.83 to 8.53. Those numbers tell you the quant drifted. They don't tell you where: which tokens, which layers, which block inside a layer, or whether the weights are even the main cause compared with the KV cache or the batch size.

I built heterodyne to answer that.

zaydmulani09 / heterodyne

Diff two llama.cpp runs token by token and layer by layer, and move activations between them to test which layers, and which attention or MLP blocks, carry the divergence. Local, CPU, no API keys.

heterodyne

Diff two llama.cpp runs token by token and layer by layer, then move activations between them to test which layers' divergence actually reaches the output.

A quantized GGUF, a quantized KV cache, a different batch size, a fine-tune: each changes what your model says, and the usual answer to "how much, and where?" is one aggregate number. heterodyne runs two configurations of llama.cpp (two arms: a GGUF file plus the runtime settings that can change its numbers) on identical tokens and shows:

• where the outputs differ: exact full-vocabulary KL(A‖B) and top-1 changes at every position, next to a measured numerical floor;
• where inside the network the difference appears: the residual stream of both arms after every layer;
• which layers carry it to the output: activation transplants that write one arm's residual stream into the other, layer by layer, in llama.cpp's own numerics;
• attention or MLP…

Why not the usual interpretability tools

TransformerLens, nnsight and the rest run PyTorch. But on CPU, llama.cpp quantizes the activations to 8 bits before every quantized matrix multiply. A PyTorch model with dequantized weights is a different model from the GGUF you actually run. And llama.cpp's own llama-perplexity --kl-divergence reports aggregates only.

So heterodyne works inside llama.cpp, in the same kernels you deploy.

Two arms, same tokens

An arm is a GGUF file plus the runtime settings that can change its numbers: K and V cache types, flash attention, ubatch size. heterodyne runs arm A and arm B on identical tokens, chunk by chunk, and records:

• exact KL(A‖B) over the full vocabulary at every position, summed in f64 from raw logits, plus every top-1 change and how confident it was;
• a numerical floor: arm A re-run with an odd batch split, as a yardstick for how big harmless differences get on this machine;
• the residual stream of both arms after every layer;
• transplants (next section);
• optionally, every combination of the settings that differ, to attribute the change to one of them.

Everything is saved as an experiment file with provenance (GGUF hashes, llama.cpp commit, build flags) and an integrity hash. heterodyne verify re-runs it and compares every stored number exactly.

The hook

llama.cpp's scheduler accepts an eval callback, cb_eval. When it's set, the scheduler asks for each graph node "do you want this tensor?" and, if you say yes, computes everything up to that node, synchronizes, and hands you the data before anything downstream runs.

A small C++ shim uses that moment to either copy the tensor out or overwrite it. Every consumer of a node comes after it in topological order, so an overwrite is seen by everything downstream. Tensors are addressed by graph name: l_out-N is the residual stream after layer N, ffn_inp-N the residual between layer N's attention and its MLP. The shim counts rows per ubatch and checks that every row of every hooked tensor was seen, so a missing name is an error, not silent zeros. Rust only sees structs I define, so a llama.cpp update can't silently shift a field under the FFI.

Transplants

Denoise: run B, but write A's l_out-N into it, and measure KL(A‖patched B). Noise: the reverse. Do it at every layer and you get a curve. For Q4_K_M vs BF16:

layer   0      4      8      12     16     20     24     26     27
KL    0.062  0.052  0.038  0.029  0.021  0.013  0.006  0.004  0.0013

It falls steadily over all 28 layers: no single layer carries the divergence. The 0.0013 left after the last layer is what the final norm and output head add on their own.

At the is_prime token, writing BF16's residual into Q4_K_M brings that token's KL from 0.48 after layer 6 down to 0.04 after layer 9, and writing Q4_K_M's residual into BF16 reproduces the flip from about layer 12. At that token, the decisive difference forms in layers 6 to 12.

New: attention or MLP?

The obvious way to split a layer is to write A's MLP output into B. The problem: that also erases the MLP's reaction to every difference upstream, so a block with identical weights in both arms can still look responsible.

So heterodyne restores A's whole residual at two points per layer, after attention (ffn_inp-N) and after the MLP (l_out-N), and takes the steps between consecutive points. Attention's share at layer N is the KL with A restored at the layer's input minus the KL with A restored after attention; the MLP's is the next step. The steps telescope: summed over all layers they equal the untouched KL minus what's left after the last layer, so nothing is counted twice.

There's a check with a known answer. If two arms differ only in K-cache type, only attention reads the K cache, so every MLP step must be exactly zero. The tests check that bitwise, and it holds on Qwen3 too.

12 chunks, 1,282 scored positions, 95% intervals over chunks:

Restoring BF16, the paired difference is 0.006 (−0.011 to 0.022): it spans zero, and attention's part ranges from 20% to 82% across chunks, so on this text the two can't be told apart. Injecting Q4_K_M, the MLPs add more (0.014, 0.005 to 0.023). The largest single step is layer 0's attention, but llama.cpp doesn't name layer 0's input, so that bar also holds any difference in the token embeddings. A few steps are negative (layer 2's attention, −0.0014): restoring that block alone left Q4_K_M further from BF16.

It costs two more passes per layer per chunk; the 12-chunk run took 29 minutes on a laptop i7-1355U.

Other things it turned up

All on Qwen3-0.6B, same text, same laptop:

• The KV cache matters more than you'd think. Same Q8_0 file twice, B with a q8_0 K and V cache: KL 0.0038. That's further than quantizing the weights from BF16 to Q8_0 (0.0022).
• llama.cpp's CPU path isn't batch-invariant at every size. With flash attention off, Q4_K_M output was bitwise identical at ubatch sizes that are multiples of 4 and changed at others (ubatch 63: 4.7% of top tokens). The repacked K-quant matmul quantizes activation rows in groups of 4 and sends the remainder down a separate path. That's why heterodyne's numerical floor uses an odd split.

What this is and isn't

• One small model, one text, one machine. Intervals are 95% t-intervals with a cluster-robust standard error over chunks, because tokens within a chunk are correlated.
• Transplants are interventions on your input. They say where restoring one model's activations removes the divergence on that text, not which block's weights quantize worse in general.
• CPU only. GPU kernels compute differently, and nothing here says anything about them yet.

Try it

Grab a binary from the releases (Linux, Windows, macOS arm64) and run:

heterodyne serve --models ~/models

or from the command line:

heterodyne diff --a Qwen3-0.6B-BF16.gguf --b Qwen3-0.6B-Q4_K_M.gguf \
  --text-file examples/corpus.txt --chunk 128 --max-chunks 12 --components

The experiment files behind every number here are in examples/. If you run it on a bigger model or different prompts, I'd love to see what you get.