TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning
TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning
TRIAGE 透過分段診斷重新平衡策略梯度更新,並對嚴重失配進行有界修復,以穩定原生 NVFP4 強化學習中的策略最佳化。Qwen3-4B 和 Qwen3-30B-A3B 的實驗在五項數學推理基準測試中達到全精度水平的表現,並在整個評估訓練期間保持穩定。配合 TRIAGE 的原生 NVFP4 推理 rollout 吞吐量較 BF16 最多提升 2.3x。
Published on Oct 5
Authors:
,
,
,
,
,
,
,
,
Abstract
Low-precision execution can substantially accelerate reinforcement learning (RL) for large language models, but discrepancies between learner and sampler execution can destabilize policy optimization. In this paper, we characterize the interaction between mismatch and the policy-gradient direction, distinguishing locally amplifying from contracting update contributions that mismatch magnitude alone cannot identify. In native NVFP4 runs, we observe an early imbalance between the two amplifying regions, favoring negative-advantage, negative-gap updates. Their tail tokens become concentrated in a small fraction of response segments before mismatch spreads globally. Motivated by these findings, we introduce TRIAGE, a direction-aware stabilization method that uses segment-level diagnosis to selectively rebalance policy-gradient updates and applies bounded repair to residual severe mismatch. TRIAGE modifies the optimization objective while retaining native NVFP4 weight-and activation 4-bit (W4A4) forward execution on both the sampler and learner. Experiments on Qwen3-4B and Qwen3-30B-A3B show stable optimization throughout the evaluated training horizon and achieve full precision level performance across five mathematical reasoning benchmarks, while native NVFP4 with TRIAGE provides up to 2.3x higher rollout throughput than BF16.
View arXiv page View PDF GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.07043
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.07043 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.07043 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.07043 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co