標準 bf16 捨入會丟棄 99% 的 RL 策略更新

If you run reinforcement learning post-training on language models using frameworks like verl, TRL, or OpenRLHF, your rollout engine is probably serving weights that are dozens of optimizer steps out of date.

The issue is not network latency or worker contention. It comes from the cast that converts optimizer master weights into the weights served by your inference engine.

Modern post-training stacks keep master weights in fp32 inside PyTorch and serve rollouts from a bf16 copy hosted in vLLM or SGLang. Every open-source framework bridges those runtimes with standard round-to-nearest conversion.

A recent Stanford paper from Andrew Gao and William Gao (arXiv:2610.41073, code at github.com/andrewgcodes/faircast) shows that at standard RL learning rates, round-to-nearest quietly discards more than 99 percent of per-step parameter updates.

The bf16 dead zone

Bfloat16 has a 7-bit significand. For a weight of magnitude |w|, the distance between adjacent representable numbers is roughly |w| / 128.

Round-to-nearest maps any update smaller than half that distance straight to zero. That creates an arithmetic dead zone of |w| / 256.

When you train with Adam at typical post-training learning rates between 1e-6 and 3e-7, almost every per-parameter update falls below that threshold. On Qwen3-1.7B, only 0.4 to 0.5 percent of served weights actually change on any given optimizer step. A weight has to accumulate updates in the same direction for 50 to 200 consecutive steps before it crosses a bf16 grid boundary and changes the served model.

If you use Muon, whose per-element updates are roughly uniform in magnitude, the problem gets worse. At 1e-6, practically none of the updates clear the rounding threshold. The inference engine is generating rollouts from a model that is effectively frozen.

Why standard diagnostics miss it

Most RL stacks track training-inference mismatch through an importance ratio between the trainer forward pass and the sampler log-probabilities. Frameworks also use PPO ratio clipping, Truncated Importance Sampling, and IcePop token masking to guard against numerical drift.

None of those tools detect this lag.

In nearly every framework, the trainer forward pass runs in bf16 autocast or evaluates the exported bf16 weights directly. Both the trainer and the sampler evaluate weights downstream of the round-to-nearest cast.

Because both runtimes evaluate the same rounded copy, their visible ratio stays close to 1.0 throughout training. In the paper's measurements on MATH and logic tasks, the visible log ratio hovered around 0.01 per token.

The ratio that actually governs policy gradient updates is between the fp32 master weights and the served bf16 copy. Frameworks never form that ratio. While the visible metric looked clean, the hidden ratio grew to between 0.10 and 0.45, and effective sample size on 2,000-token responses dropped to between 0.2 and 0.35.

The saved checkpoint discrepancy

This rounding behavior also creates a gap between what you train and what you save.

At checkpoint time, frameworks save the fp32 master weights from the optimizer. On knights-and-knaves and zebra puzzles, the paper found that the saved fp32 checkpoint scored between 12 and 50 points lower than the bf16 model that actually generated the rollouts during the run.

If you take that saved fp32 checkpoint and convert it to fp16 or FP8 for production serving, your deployed model does not match the behavior of the model you evaluated during training.

The one-line fix: Fair Cast

The paper proposes Fair Cast, which fixes the problem with two changes:

1. Replace round-to-nearest with stochastic rounding at the export barrier. When an fp32 weight falls between two bf16 numbers, round up with probability proportional to its fractional position. In expectation, the exported weights match the master weights exactly. Implementation requires adding 16 random bits to the fp32 representation before truncating. On a 1.7B parameter model, this takes 22 to 38 milliseconds.
2. Evaluate training loss on the exact served copy, ensuring the differentiated policy matches the policy generating rollouts.

Across 25 matched seed pairs on Qwen3 models from 0.6B to 4B parameters, Fair Cast beat round-to-nearest on 18 runs, tied on 6, and trailed on 1. On zebra logic puzzles, where round-to-nearest collapsed to 0.33 accuracy, Fair Cast reached 0.57.

What to check in your stack

If you run RLVR, PPO, or GRPO on open-source frameworks, inspect the call that syncs weights from your PyTorch optimizer module to your inference workers.

If the sync uses a standard PyTorch cast like weight.to(torch.bfloat16), it is running round-to-nearest. At learning rates below 5e-6, your rollout engine is serving weights that lag your optimizer by dozens of steps.

Switching to stochastic rounding at the export call takes minimal code and adds negligible latency. If your reasoning runs plateau early while your loss curves look healthy, check your weight-export cast before tuning your rewards.