跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 27 天前AI 評分36

S3AC:面向 LLM 推理的穩定、選擇性且可擴展的 Actor-Critic 方法

S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning

AI 導讀

S3AC 提出一種用於 LLM 長鏈式推理的 Actor-Critic 訓練方法,透過難度感知提示詞採樣,在不依賴分組 rollout 的情況下改善訓練。它結合 token 級優勢估計與提示詞級基線,在 1.5B 至 397B 參數的稠密及混合專家模型上提升分組 rollout 表現,勝過 GRPO-X;在 35B 規模下亦可持續單次 rollout 訓練。

正文

Published on Oct 26

Authors:

,

,

,

,

Abstract

S3AC introduces a stabilized actor-critic training method with difficulty-aware prompt sampling to improve long reasoning in large language models without relying on grouped rollouts.

Critic-free methods are widely used in reinforcement learning with verifiable rewards (RLVR) for long chain-of-thought (CoT) reasoning, avoiding difficult and costly critic training. Common alternatives lack token-level credit assignment and typically rely on grouped rollouts to estimate prompt-specific baselines. A critic can address both limitations, but long-CoT critic training is hindered by insufficient coverage of trajectories with diverse lengths and early gradients dominated by the newly initialized value head. We introduce S3AC (Stabilized, Selective, and Scalable Actor-Critic) for LLM reasoning, built around a practical critic-training recipe. The trained critic provides token-specific advantages and prompt-specific baselines. Using the prompt-level baseline to estimate difficulty, we propose Difficulty-Aware Prompt Sampling (DAPS). DAPS combines this estimate with outcome history to select prompts before generation, reducing rollout waste without offline difficulty initialization. S3AC improves group-rollout performance across dense and mixture-of-experts models from 1.5B to 397B parameters, outperforming a modern GRPO variant (GRPO-X). At 35B, it sustains single-rollout training while tested critic-free baselines deteriorate after initial gains.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.35518

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.35518 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.35518 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.35518 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co