Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL

• URL: http://arxiv.org/abs/2605.24001v2
• Date: Tue, 26 May 2026 04:35:31 GMT
• Title: Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL
• Abstract summary: One-step text-to-image generation has enabled real-time synthesis with remarkable efficiency and quality.<n>Previous reinforcement learning methods for one-step generators combine image-space reward optimization with diffusion noisy-space distribution matching.<n>We propose Diff-Instruct with Diffused Reward (DIDR), a data-free trajectory-level alignment framework derived from KL minimization.
• Score: 16.321536102397733
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in one-step text-to-image generation have enabled real-time synthesis with remarkable efficiency and quality. Previous reinforcement learning methods for one-step generators combine image-space reward optimization with diffusion noisy-space distribution matching. This paradigm brings challenges due to a mismatch between terminal reward optimization and the underlying generative dynamics. As a result, optimization tends to exploit stochastic degrees of freedom, often improving reward at the expense of image fidelity. To address this issue, we propose Diff-Instruct with Diffused Reward (DIDR), a data-free trajectory-level alignment framework derived from Integral KL minimization. DIDR propagates the RLHF-optimal reward-tilted clean-image distribution across all noise levels along the diffusion trajectory. We show that this objective admits the same minimizer as clean-image RLHF, while naturally inducing the Diffused Reward Score (DRS), which acts as a reward-driven correction to the reference score function. To make this practical, we further introduce the Diffused Reward Proxy (DRP), an efficient estimator of DRS based on differentiable short-step denoising. Extensive experiments demonstrate that DIDR consistently Pareto-dominates existing one-step SDXL baselines. Moreover, when transferred to a 6B DiT backbone (Z-Image), DIDR surpasses its 50-step teacher in preference alignment while requiring only a single generation step.

Related papers

• Latent Reward Registers for Diffusion Preference Alignment [46.49072106092868]
  We propose Latent Reward Registers, a mechanism that estimates terminal preference directly from intermediate noisy latents.<n>This independent readout mechanism extracts latent reward evidence without altering the generator's hidden states or velocity field.<n>For training, Reward-Gradient On-Policy Distillation (RG-OPD) distills reward-guided updates along on-policy trajectories, bypassing the computationally expensive rollouts of standard policy gradients.<n>For inference, Reward-Guided Sampling (RGS) steers trajectories via magnitude-matched reward gradients without parameter updates.
  arXiv  Detail & Related papers  (2026-08-04T17:00:52Z)
• Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models [73.08789211016567]
  Diffusion LAIR is a reward-aware listwise preference optimization method for diffusion models.<n>Experiments show it outperforms strong preference optimization baselines on text-to-image generation, compositional generation, and image editing benchmarks.
  arXiv  Detail & Related papers  (2026-05-26T03:09:24Z)
• Reinforcing Few-step Generators via Reward-Tilted Distribution Matching [34.875805803270524]
  We propose Reward-Tilted Distribution Matching Distillation (RTDMD)<n>We show that minimizing KL divergence to a reward-tilted teacher distribution naturally decomposes into a distribution matching term and a reward term.<n>Experiments on SD3, SD3.5, and FLUX.2 demonstrate that RTDMD establishes new state-of-the-art results across preference, aesthetic, and compositional metrics.
  arXiv  Detail & Related papers  (2026-05-25T17:59:21Z)
• Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models [56.67321805551389]
  Reinforcement learning (RL) has become an effective way to improve prompt alignment and perceptual quality in diffusion and flow-matching generators.<n>The sampler, controlling the exploration behavior and denoising dynamics, is thus part of the policy.<n>We propose a new sampler that balances effective exploration with stability.
  arXiv  Detail & Related papers  (2026-05-22T11:37:22Z)
• Disentangling Generation and Regression in Stochastic Interpolants for Controllable Image Restoration [30.480465453951982]
  DiSI is a unified framework that Disentangles the Interpolant process into independent generation and regression components.<n>We show that DiSI efficiently achieves competitive results on various Image Restoration tasks, while uniquely offering the inference-time flexibility to control the distortion-perception trade-off within a single model.
  arXiv  Detail & Related papers  (2026-05-20T16:41:32Z)
• Noise-Started One-Step Real-World Super-Resolution via LR-Conditioned SplitMeanFlow and GAN Refinement [32.05719533625606]
  We propose a noise-started one-step Real-ISR framework via LR-conditioned SplitMeanFlow and GAN refinement.<n> SMFSR achieves state-of-the-art quality among one-step diffusion-based Real-ISR methods while retaining fast single-step inference.
  arXiv  Detail & Related papers  (2026-05-10T05:01:32Z)
• LPNSR: Prior-Enhanced Diffusion Image Super-Resolution via LR-Guided Noise Prediction [4.609499769793957]
  Diffusion-based image super-resolution (SR) aims to reconstruct high-resolution (HR) images from corresponding low-resolution (LR) observations.<n>The state-of-the-art residual-shifting diffusion framework efficient 4-step inference suffers from severe degradation performance in compact sampling trajectories.<n>We propose a prior-enhanced efficient diffusion framework to address these issues.
  arXiv  Detail & Related papers  (2026-03-22T03:52:38Z)
• Diffusion Reinforcement Learning via Centered Reward Distillation [35.979608265594685]
  We present textbf Reward Distillation (CRD), a diffusion RL framework derived from KL-regularized reward models built on forward-process fine-tuning.<n>To enable reliable text-to-image fine-tuning, we introduce techniques that explicitly control distribution drift.<n>Experiments on text-to-image post-training with textttGenEval and textttOCR rewards show that competitive SOTA reward optimization results with fast convergence and reduced reward hacking preference.
  arXiv  Detail & Related papers  (2026-03-14T21:29:33Z)
• MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Models [86.07486858219137]
  Diffusion models excel at generating images conditioned on text prompts.<n>The resulting images often do not satisfy user-specific criteria measured by scalar rewards such as Aesthetic Scores.<n>Recently, inference-time alignment via noise optimization has emerged as an efficient alternative.<n>We show that this approach suffers from reward hacking, where the model produces images that score highly, yet deviate significantly from the original prompt.
  arXiv  Detail & Related papers  (2025-10-02T00:47:36Z)
• Flow-GRPO: Training Flow Matching Models via Online RL [80.62659379624867]
  We propose Flow-GRPO, the first method to integrate online policy reinforcement learning into flow matching models.<n>Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation into an equivalent Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps.
  arXiv  Detail & Related papers  (2025-05-08T17:58:45Z)
• Reconciling Stochastic and Deterministic Strategies for Zero-shot Image Restoration using Diffusion Model in Dual [47.141811103506036]
  We propose a novel zero-shot image restoration scheme dubbed Reconciling Model in Dual (RDMD)<n>RDMD uses only a bftextsingle pre-trained diffusion model to construct texttwo regularizers.<n>Our proposed method could achieve superior results compared to existing approaches on both the FFHQ and ImageNet datasets.
  arXiv  Detail & Related papers  (2025-03-03T08:25:22Z)
• Improved Distribution Matching Distillation for Fast Image Synthesis [54.72356560597428]
  We introduce DMD2, a set of techniques that lift this limitation and improve DMD training. First, we eliminate the regression loss and the need for expensive dataset construction. Second, we integrate a GAN loss into the distillation procedure, discriminating between generated samples and real images.
  arXiv  Detail & Related papers  (2024-05-23T17:59:49Z)
• BlindDiff: Empowering Degradation Modelling in Diffusion Models for Blind Image Super-Resolution [52.47005445345593]
  BlindDiff is a DM-based blind SR method to tackle the blind degradation settings in SISR. BlindDiff seamlessly integrates the MAP-based optimization into DMs. Experiments on both synthetic and real-world datasets show that BlindDiff achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-03-15T11:21:34Z)
• SinSR: Diffusion-Based Image Super-Resolution in a Single Step [119.18813219518042]
  Super-resolution (SR) methods based on diffusion models exhibit promising results. But their practical application is hindered by the substantial number of required inference steps. We propose a simple yet effective method for achieving single-step SR generation, named SinSR.
  arXiv  Detail & Related papers  (2023-11-23T16:21:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.