跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 17 天前AI 評分35

FewStepRL:以強化學習微調少步擴散模型

FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning

AI 導讀

FewStepRL 將強化學習目標調整至少步擴散模型,透過轉換至 x-space、梯度分解及錨點調整,建立多種方法的少步版本。實驗顯示,獎勵可微且其梯度成本可承受時,獎勵梯度方法效果最佳;否則,DiffusionNFT 類方法通常能以較少迭代取得高獎勵。額外推理步數整體未帶來一致收益,但部分輸出在採樣步數增至訓練時的 2 至 4 倍後有所改善。

正文

Published on Nov 5

Authors:

,

,

,

,

,

Abstract

Reinforcement learning objectives for diffusion models are reformulated for few-step samplers through x-space transformation, gradient decomposition, and anchor adaptation, yielding modular few-step variants of several methods and showing reward-gradient methods excel with differentiable rewards while DiffusionNFT-style methods achieve high rewards in fewer iterations.

Reinforcement learning (RL) has become a standard post-training method for diffusion models, but most RL objectives roll out the current policy at every iteration, and with a many-step sampler these rollouts typically dominate the cost of training. Few-step models cut this cost, yet they model an average velocity over an interval, whereas existing RL objectives are built for an instantaneous velocity with a many-step sampler. We adapt RL objectives to few-step models systematically in three phases, namely a transformation, a decomposition and an adaptation. The transformation to x-space expresses each objective we consider through a common policy prediction, at the clean end by default and at an explicitly marked level for transition or flow-map objectives, so that objectives stated over densities, velocities or rewards become comparable. The decomposition writes the gradient of each of these objectives as the sum of a self-regression toward a reference and a reward augmentation, so that a method reduces, at the gradient level, to a choice of anchors and weights. The adaptation computes each anchor from a few-step model at the end of its transport or along its tangent, with the instantaneous velocity recovered from the network itself or from an auxiliary network. This yields few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD and ReFL in one modular codebase. In our experiments, reward-gradient-based methods are the most effective when the reward is differentiable and its gradient is affordable; otherwise DiffusionNFT-style methods generally reach high rewards in relatively few iterations despite a sometimes higher per-iteration cost. Extra inference steps show no systematic aggregate benefit in our experiments, although selected outputs visibly improve with two to four times as many sampling steps as in training.

View arXiv page View PDF Project page GitHub 2 Add to collection

Get this paper in your agent:

hf papers read 2611.08240

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 2

Delcher/fewsteprl-sd35

Text-to-Image •

Updated 1 day ago

Delcher/fewsteprl-wan21

Text-to-Video •

Updated 1 day ago

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.08240 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.08240 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co