LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches
LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches
LoGRA 以低秩梯度草圖保留有效學習訊號,降低大型語言模型強化學習後訓練的記憶體需求。在推理任務中,LoGRA 平均可減少最多 45.7% 的訓練記憶體用量,且不犧牲效能。它亦可在單一八 GPU 節點上,穩定訓練 27B 參數模型超過 1,100 步;使用 dense Adam 則會耗盡記憶體。
Published on Oct 5
Authors:
,
,
,
,
,
,
Abstract
Reinforcement learning (RL) has greatly advanced the capabilities of large language models (LLMs), but its memory demands remain a barrier to broader adoption. We introduce LoGRA, an approach to RL post-training that reduces memory by retaining useful learning signals in low-rank gradient sketches. These compact representations support both model updates and efficient policy synchronization. To prevent overly large updates from disrupting learning, we complement gradient compression with predicted-KL step control, which estimates policy changes before applying each update and adjusts its magnitude accordingly. Across reasoning tasks, LoGRA reduces average training memory by up to 45.7\% without sacrificing performance. It also enables stable training of a 27B-parameter model for over 1,100 steps on a single eight-GPU node, where dense Adam runs out of memory, making previously memory-infeasible RL training practical. Code is available in the https://github.com/skzhang1/labs-molt/tree/logra/examples/scripts/logra{Molt library}.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2610.06647
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.06647 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.06647 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.06647 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co