跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-05AI 評分48

LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches

LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches

AI 導讀

LoGRA 以低秩梯度草圖保留有效學習訊號,降低大型語言模型強化學習後訓練的記憶體需求。在推理任務中,LoGRA 平均可減少最多 45.7% 的訓練記憶體用量,且不犧牲效能。它亦可在單一八 GPU 節點上,穩定訓練 27B 參數模型超過 1,100 步;使用 dense Adam 則會耗盡記憶體。

正文

Published on Oct 5

Authors:

,

,

,

,

,

,

Abstract

Reinforcement learning (RL) has greatly advanced the capabilities of large language models (LLMs), but its memory demands remain a barrier to broader adoption. We introduce LoGRA, an approach to RL post-training that reduces memory by retaining useful learning signals in low-rank gradient sketches. These compact representations support both model updates and efficient policy synchronization. To prevent overly large updates from disrupting learning, we complement gradient compression with predicted-KL step control, which estimates policy changes before applying each update and adjusts its magnitude accordingly. Across reasoning tasks, LoGRA reduces average training memory by up to 45.7\% without sacrificing performance. It also enables stable training of a 27B-parameter model for over 1,100 steps on a single eight-GPU node, where dense Adam runs out of memory, making previously memory-infeasible RL training practical. Code is available in the https://github.com/skzhang1/labs-molt/tree/logra/examples/scripts/logra{Molt library}.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.06647

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.06647 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.06647 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.06647 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co