跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-15AI 評分37

Routing Policy Optimization for Online Diffusion Reinforcement Learning

Routing Policy Optimization for Online Diffusion Reinforcement Learning

AI 導讀

Routing Policy Optimization(RPO)在擴散生成器內引入隨機注意力路由,以獎勵訊號對齊生成器,同時保留原有的確定性採樣器。文字生成圖像和文字生成影片的實驗顯示,RPO 改善了生成質素,訓練效率較 Flow-GRPO 類方法高近 4x。

正文

Published on Oct 15

Authors:

,

,

,

,

,

,

,

,

,

Abstract

Routing Policy Optimization introduces stochastic attention routing inside diffusion generators to align them with reward signals while preserving the original deterministic sampler and improving training efficiency.

Reinforcement learning can align visual generators with reward signals beyond their pretraining objectives, but existing score-function methods typically introduce stochasticity by perturbing the reverse denoising trajectory, leading to costly rollout-based optimization. We instead ask whether stochastic exploration can be introduced inside the generator while preserving its original sampler. Routing Policy Optimization (RPO) answers this question by turning attention routing into a stochastic exploration action. At a few selected denoising levels, each query block keeps its own block and samples further key blocks in proportion to the generator's native attention mass, or keeps dense attention. The sampled route perturbs the velocity of that transition, and a Gaussian likelihood around the dense velocity scores the perturbation with a score-function objective. RPO further uses a controlled two-rollout estimator: two rollouts share the same prompt, initial latent, routed denoising levels, and dense anchor layers, differing only in their sampled attention routes. Their reward difference provides an advantage signal for the routed transitions. Because the likelihood differentiates the dense velocity itself, the learned updates remain effective after routing is removed. At inference, RPO restores dense attention and retains the original deterministic sampler. Experiments on text-to-image and text-to-video generation show that RPO improves generation quality while achieving nearly 4x higher training efficiency than Flow-GRPO-style methods. Please see our project page at https://velox-world.com/rpo for visual results.

View arXiv page View PDF Project page Add to collection

Get this paper in your agent:

hf papers read 2610.22282

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.22282 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.22282 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.22282 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co