量化:低精度權重為何能維持全精度準確度
Quantisation: Why Low-Precision Weights Keep Full-Precision Accuracy
模型量化以 4-bit 權重取代 32-bit 權重,可將模型縮小八倍,通常在相同基準測試中只比原模型低一至兩分。每組量化有助保留準確度,但激活值中的少數極端值會令直接量化失效;SmoothQuant、AWQ 和 GPTQ 會針對量化誤差作出調整。
Quantisation: Why Low-Precision Weights Keep Full-Precision Accuracy
A modern language model stores tens of billions of weights, each one a 32-bit floating point number. Cut every weight to 4 bits and you shrink the model eightfold, halve the memory bandwidth, and run it on hardware that could never hold the original. The surprise is not that this works at all. The surprise is how little accuracy it costs. A 4-bit model usually scores within a point or two of its 32-bit parent on the same benchmarks, and sometimes the gap is smaller than the noise between two training runs.
That should feel wrong. Throwing away seven eighths of the information in every parameter ought to break something. It mostly does not, and the reasons why reveal something important about what neural networks actually store in their weights.
What quantisation actually does
Quantisation replaces a continuous weight with a value from a small, finite set. The standard scheme is affine quantisation. For a weight tensor, pick a scale s and a zero point z , then map each weight w to an integer code:
q=clamp(round(ws)+z,qmin,qmax)
The dequantised weight used at inference time is:
w^=s(q−z)
For 8-bit quantisation, q runs over 256 values; for 4-bit, only 16. The scale is usually chosen from the observed range of the weights, s=(wmax−wmin)/(2b−1) for b bits, and the zero point shifts the grid so that the value zero is represented exactly, which matters because many weights and activations sit near zero.
Two design choices dominate everything that follows. Symmetric quantisation sets z=0 and a single scale for positive and negative values, which is cheap in hardware but wastes grid points when the distribution is skewed. Asymmetric quantisation keeps a nonzero zero point and fits the grid to the actual range, which costs a little extra arithmetic but represents skewed distributions far better.
The second choice is granularity. A single scale per tensor is simple; a scale per channel or per group of, say, 128 weights adapts to the fact that different channels live on wildly different scales. Per-group quantisation is the main reason modern 4-bit LLM methods work as well as they do: the grid is fitted locally, where the weight distribution is tight, instead of globally, where a few large weights stretch the grid and crush everything else.
Why the damage is so small
The first reason is that quantisation error behaves like noise, and networks are already trained to be robust to noise. If the quantisation step s is small relative to the spread of the weights, the error w^−w is roughly uniform on [−s/2,s/2] , with variance:
Var(w^−w)≈s212
A layer with thousands of inputs sums thousands of these small, roughly independent errors. By the same averaging that makes a Monte Carlo estimate converge, the relative error in the layer output shrinks as the fan-in grows. Wide layers quantise better than narrow ones, for exactly this reason.
The second reason is redundancy. Trained networks are massively overparameterised, and many directions in weight space barely affect the loss. The loss landscape around a trained minimum is flat in most directions and curved in only a few. Quantisation perturbs the weights in a random direction, which is overwhelmingly likely to point along the flat directions. This is the same flat-minimum intuition behind sharpness-aware minimisation: the network does not need every weight to be exactly right, it needs the combination to stay inside a broad, forgiving basin.
The third reason is that the quantities that matter are not the weights themselves but the outputs they produce. A weight that is rarely activated, or that feeds into a neuron whose output is saturated, can be approximated crudely with no visible effect. Quantisation error only matters where it changes a decision, and most weights sit far from any decision boundary.
Weights are easy, activations are hard
So far this sounds like a free lunch, and for weights it nearly is. Post-training quantisation of weights alone, with per-group scales and 4 bits, routinely preserves accuracy on language models with no retraining at all.
Activations are a different story. Weight distributions are tame, roughly bell-shaped and stable. Activation distributions are not. A handful of channels in a transformer produce activations hundreds of times larger than the rest, the so-called outlier features. A single global scale fitted to the activation range is then dominated by those outliers, and every ordinary activation collapses onto a couple of grid points. This is why naive 8-bit activation quantisation often fails badly while 8-bit weight quantisation works fine.
The fixes all attack the outliers. SmoothQuant shifts the difficulty from activations to weights by rescaling channels before quantisation, exploiting the fact that a linear layer is invariant to per-channel rescaling of its inputs and weights. AWQ, activation-aware weight quantisation, protects the small subset of weights that interact with large activations, keeping them at higher precision while the rest go to 4 bits. GPTQ takes a different route: it quantises weights one column at a time and compensates the remaining unquantised weights for the error introduced so far, using second-order information from the Hessian of the layer. All three methods share the same insight: quantisation error is not uniform, so the budget should be spent where the error hurts most.
Post-training quantisation versus quantisation-aware training
There are two broad regimes. Post-training quantisation (PTQ) takes a finished model and quantises it directly, using a small calibration set to choose scales. It is fast and it is usually enough for 8-bit weights and activations, and for 4-bit weights with the methods above.
Quantisation-aware training (QAT) inserts fake quantisation into the training loop: weights are quantised in the forward pass and dequantised before use, so the network learns to be robust to the rounding. The gradient through the rounding operation does not exist, so QAT uses the straight-through estimator, passing the gradient through unchanged as if rounding were the identity. This is a biased approximation, but it works in practice because the network adapts around it. QAT is what makes 4-bit activations and very low bit-widths, like 2-bit weights, feasible, at the cost of a retraining run.
The practical rule is simple: try PTQ first. If the accuracy gap is within tolerance, ship it. Reach for QAT only when the bit-width or the activation distribution forces your hand.
A minimal example
The mechanics are easy to see in PyTorch. Here is a small fake-quantisation experiment on a single linear layer, showing how much error 4-bit per-group quantisation actually introduces:
import torch
torch.manual_seed(0)
W = torch.randn(512, 512) * 0.05 # a typical weight matrix
x = torch.randn(64, 512) # a batch of inputs
def quantize_groupwise(W, bits=4, group=128):
Wg = W.view(-1, group)
wmin, wmax = Wg.min(dim=1, keepdim=True).values, Wg.max(dim=1, keepdim=True).values
qmax = 2 ** bits - 1
s = (wmax - wmin) / qmax
q = torch.round((Wg - wmin) / s).clamp(0, qmax)
return (q * s + wmin).view_as(W)
Wq = quantize_groupwise(W)
y, yq = x @ W.T, x @ Wq.T
rel_err = (yq - y).norm() / y.norm()
print(f"Weight error: {(Wq - W).norm() / W.norm():.4f}")
print(f"Output error: {rel_err:.4f}")
The weight error looks large, several percent, but the output error is far smaller, because the errors partly cancel across the 512 inputs. That gap between weight error and output error is the whole story of quantisation in one experiment. If you repeat it with a single global scale instead of per-group scales, the output error jumps several fold, which is why granularity matters more than bit-width.
Limitations and common pitfalls
Quantisation is not free, and the failures are instructive.
First, outliers break naive schemes. If your model has a few channels with extreme activations, per-tensor activation quantisation will fail, and you need one of the outlier-aware methods or per-channel scales. Always inspect the activation ranges before choosing a scheme.
Second, error accumulates across layers. A 1 percent error per layer is harmless in isolation, but a 100-layer network compounds it. Deep networks and long generation horizons are where low-bit quantisation finally bites, and the last layers, closest to the output, are usually the most sensitive.
Third, calibration data matters. PTQ scales are fitted on a calibration set, and if that set does not match deployment data, the scales are wrong. A calibration set of generic text will miscalibrate a model deployed on code. Match the calibration distribution to the real one.
Fourth, quantisation interacts with training dynamics. Quantising during fine-tuning without accounting for it, or quantising a model that was trained with heavy weight decay, can behave unexpectedly, because the weight distribution itself was shaped by the optimiser. And a model quantised to 4 bits cannot be fine-tuned further in the usual way, since the gradients are smaller than the quantisation step and round to zero.
Finally, the hardware has to support the format. A 4-bit weight saves memory only if the inference engine can actually compute with 4-bit values efficiently; otherwise the dequantisation overhead eats the gain. Check the kernel support for your deployment target before committing to a bit-width.
The takeaway
Quantisation works because neural networks store information redundantly, in flat basins of the loss landscape, and because thousands of small, independent rounding errors average out in a wide layer. The art is in spending the precision budget where it matters: per-group scales for weights, outlier-aware methods for activations, and quantisation-aware training when the bit-width gets aggressive. The result is models that run eight times smaller and several times faster, with accuracy so close to the original that the difference is often within experimental noise. The weights were never as precise as their 32 bits suggested; quantisation just stops pretending otherwise.
Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial
來源:DEV Community · dev.to