MuonGS 為何在 LLM 預訓練中勝過 AdamW 和 Muon
Why MuonGS Beats AdamW and Muon in LLM Pretraining
MuonGS 在 400M dense 和 1B 參數 MoE 架構上，達到相同驗證困惑度的速度比 MuonH 快 15%。它透過 Gram-Schmidt 投影控制權重範數增長，並以原位張量運算執行，無需額外配置 VRAM。此方法適用於具平滑尺度不變性的層；token embeddings 和語言模型輸出頭仍需其他調度器或 AdamW 分支。
Standard AdamW and vanilla Muon waste millions of GPU hours during LLM pretraining because uncontrolled weight-norm growth throttles the effective learning rate. We cannot keep burning H100 clusters on sub-optimal trajectory dynamics when Gram-Schmidt projection gives us direct mathematical control over weight exploration.
Here is how MuonGS fundamentally shifts pretraining economics:
- 15% Token-Equivalent Compute Reduction: On 400M dense and 1B parameter MoE architectures, MuonGS reached matched validation perplexity 15% faster than MuonH, slashing hundreds of GPU cluster hours.
- Strict Norm Growth Bounds ($O(\sqrt{t})$): By removing the radial update component via Gram-Schmidt projection, the Frobenius norm $|W_t|_F$ grows at most sublinearly instead of stalling accumulation logarithmically, guaranteeing sustained exploration ($C_T = \Omega(\sqrt{T})$).
- Zero Additional VRAM Allocation: Unlike heavy second-order optimizers that demand massive auxiliary state matrices, MuonGS executes purely in-place tensor operations with negligible step latency impact inside standard CUDA cgroups.
The one real trade-off is architectural sensitivity: Gram-Schmidt projection assumes smooth scale-invariant layers. Non-normalized components like token embeddings and language model output heads still require segregated coordinate-wise schedulers or traditional AdamW branches.
The Question to You: What is your team using for foundation pretraining and long-horizon fine-tuning in 2026? Are you still defaulting to AdamW, or running Muon and SOAP in production? Drop your stack below.
Bohdan Hordii // Systems & Sovereign Infrastructure Architect | bohdanhordii.dev
來源：DEV Community · dev.to