隨機捨入與最接近值捨入:隨機性何時能保留平均值
Stochastic Rounding vs Round-to-Nearest: When Randomness Preserves the Average
低精度神經網絡訓練中,最接近值捨入會把小於半個量化步長的更新捨為零,令微小梯度難以累積;隨機捨入則按數值與相鄰可表示值的距離決定捨入機率,讓期望值保持準確。儲存訓練完成的模型適合使用最接近值捨入;累積低精度下的小幅更新則適合使用隨機捨入。
Training a neural network in low precision means every weight update must be snapped to the nearest representable value. The obvious rule, round to nearest, is accurate for any single number and completely predictable. It also has a hidden failure: an update smaller than half a step rounds to zero and never lands, so a small gradient applied many times can be lost forever. Stochastic rounding replaces the deterministic choice with a random one that preserves the average, and the difference shows up most in long training runs at low precision.
Round-to-nearest is deterministic and locally accurate but systematically drops updates smaller than half a quantization step, while stochastic rounding chooses between neighbours with probability proportional to distance so the expected value stays exact and tiny changes survive.
The two rules produce identical-looking numbers in isolation. Over millions of updates they produce different models.
Part One: What is being compared
Round-to-nearest is what everyone learns first. A value halfway between two representable numbers goes to whichever is closer, with a tie rule, and the error on any single rounding is at most half a step. For storing a trained model, that is exactly what you want: minimal distortion of the value you have. It is also cheap, needing only a comparison.
The problem appears during accumulation. Suppose a weight is represented with a small step and the gradient each batch changes it by a tenth of that step. Round to nearest snaps every one of those changes back to the original value, so ten steps produce no movement at all. The gradient signal is not small in aggregate; it is simply below the resolution of the rule at each step.
Stochastic rounding maps a value between two neighbours to the upper one with probability equal to its fractional distance, so a value nine-tenths of the way up rounds up ninety percent of the time — the expected rounded value equals the original, and a one-tenth-step update accumulates over ten draws on average.
Part Two: The mechanism
Stochastic rounding makes the update probabilistic. A value that sits twenty percent of the way from the lower neighbour to the upper one is rounded up twenty percent of the time and down the other eighty. The result on any single step is noisier than round to nearest, but the expected value is exactly the number you started with, so a stream of tiny updates nudges the weight in the right direction over time.
The mechanism is unbiasedness under accumulation. Round to nearest is unbiased in the sense that its error is symmetric for random inputs, but for the specific case of repeated small increments it is systematically biased toward zero movement. Stochastic rounding removes that bias by construction, which is why it is used in low-precision training, in gradient accumulation and in some forms of quantised optimisation. The cost is a random number per rounding, though hardware support and cheap pseudo-random generators make this inexpensive on modern accelerators.
Part Three: Where it shows up
An everyday picture is filling a jar with a spoon when the scale only displays whole grams. Round to nearest means each fraction of a gram vanishes from the display, so after many spoonfuls the jar looks unchanged. Stochastic rounding means you occasionally record a whole gram so that, on average, the displayed total tracks what is actually in the jar. One reading is noisier; the long-run total is honest.
This matters in practice for fine-tuning on small datasets, where gradients are small and learning rates are low. A team that halves the numeric precision to fit a bigger model may find training stalls not because the model is too small but because every update rounds away, and switching the rounding rule recovers learning without changing the model or the data.
Use round-to-nearest to store a finished model with minimal distortion; use stochastic rounding when you are accumulating small changes in low precision, because preserving the expected value matters more there than getting any single step exactly right.
What is actually transferable
The lesson extends beyond neural networks. Any system that repeatedly adds quantities smaller than its representation step faces the same choice, and the fix is usually to make rounding unbiased over time rather than accurate at each instant.
Precision is not just about how many bits you have; it is about what your rounding rule does to the small things you keep adding.
Market sizing check
AI accelerators and training hardware are a market measured in tens of billions of dollars per year, and low-precision training is central to their economics because it determines how many models fit on a chip. A rounding rule that makes four-bit training converge reliably can effectively increase usable compute for a large share of that installed base, which is why a seemingly small algorithmic detail attracts hardware attention.
These are order-of-magnitude estimates from published market ranges and stated assumptions, not a forecast for any vendor.
The comparison that matters
Round-to-nearest is unbiased per number but loses small updates in low precision; stochastic rounding keeps the expected value correct and lets tiny gradients accumulate over many steps.
Creating this takes real effort — thank you for the support.
HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
來源:DEV Community · dev.to