Reinforcing Few-step Generators via Reward-Tilted Distribution Matching

• URL: http://arxiv.org/abs/2605.26108v2
• Date: Thu, 28 May 2026 17:54:16 GMT
• Title: Reinforcing Few-step Generators via Reward-Tilted Distribution Matching
• Abstract summary: We propose Reward-Tilted Distribution Matching Distillation (RTDMD)<n>We show that minimizing KL divergence to a reward-tilted teacher distribution naturally decomposes into a distribution matching term and a reward term.<n>Experiments on SD3, SD3.5, and FLUX.2 demonstrate that RTDMD establishes new state-of-the-art results across preference, aesthetic, and compositional metrics.
• Score: 34.875805803270524
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in few-step diffusion distillation have enabled efficient image generation, yet aligning these models with human preferences remains challenging. We propose Reward-Tilted Distribution Matching Distillation (RTDMD), a two-stage framework that unifies distribution matching distillation with reward-guided reinforcement learning for few-step flow generators. We show that minimizing the KL divergence to a reward-tilted teacher distribution naturally decomposes into a distribution matching term and a reward maximization term. In the first stage, we introduce Ambient-Consistent Distribution Matching Distillation (AC-DMD), which performs subinterval-wise distribution matching and augments the fake score objective with a consistency regularizer to help the fake score model track the shifting generator distribution under limited updates. In the second stage, we jointly optimize both terms: for the reward maximization term, we derive a hybrid policy gradient that combines a GRPO-style estimator for the stochastic intermediate transitions with direct reward backpropagation through the deterministic final step, and further introduce step-subset GRPO (SubGRPO) to reduce variance. Experiments on SD3, SD3.5, and FLUX.2 demonstrate that RTDMD establishes new state-of-the-art results across preference, aesthetic, and compositional metrics with only 4 inference steps, outperforming previous few-step text-to-image generation methods. Code and models are available at https://github.com/Harahan/RTDMD.

Related papers

• Joint Alignment and Distillation for Video Generation via Sample-Guided Distribution Matching [36.00855152229835]
  We propose a unified, single-stage optimization framework grounded in Distribution Matching.<n>In the standard DM framework, distillation updates the model via a gradient direction that minimizes the gap between the real and fake models.<n>We introduce DM-Align, which derives a complementary gradient direction to guide the model toward human-preferred samples.
  arXiv  Detail & Related papers  (2026-09-03T04:46:46Z)
• Constrained Diffusion Models with Primal-Dual Inference [76.64466170898369]
  We develop constrained diffusion models with primal-dual inference (PDI)<n>PDI infers the optimal primal distribution and its parametrizing dual variable.<n>We evaluate PDI on constrained sampling from a mixture of Gaussians, wireless resource allocation, and portfolio management.
  arXiv  Detail & Related papers  (2026-06-15T18:32:33Z)
• Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL [9.05769414150212]
  One-step text-to-image generation has enabled real-time synthesis with remarkable efficiency and quality.<n>Previous reinforcement learning methods for one-step generators combine image-space reward optimization with diffusion noisy-space distribution matching.<n>We propose Diff-Instruct with Diffused Reward (DIDR), a data-free trajectory-level alignment framework derived from KL minimization.
  arXiv  Detail & Related papers  (2026-05-18T21:26:33Z)
• Continuous-Time Distribution Matching for Few-Step Diffusion Distillation [57.28746398500951]
  We introduce Continuous-Time Distribution Matching (CDM), migrating the DMD framework from discrete anchoring to continuous optimization.<n>CDM achieves this through two continuous-time designs. First, we replace the fixed discrete schedule with a dynamic continuous schedule of random length.<n>Second, we propose a continuous-time alignment objective that performs active off-trajectory matching on latents extrapolated via the student's velocity field.
  arXiv  Detail & Related papers  (2026-05-07T14:56:39Z)
• Guiding Distribution Matching Distillation with Gradient-Based Reinforcement Learning [41.982957134224904]
  Diffusion distillation has shown great promise in few-step generation but often sacrifices quality for sampling speed.<n>We propose GDMD, a novel framework that redefines the reward mechanism by prioritizing distillation gradients over raw pixel outputs.<n>Our models outperform the quality of their multi-step teacher and substantially exceed previous DMDR results in GenEval and human-preference metrics.
  arXiv  Detail & Related papers  (2026-04-21T02:57:13Z)
• Effective Test-Time Scaling of Discrete Diffusion through Iterative Refinement [51.54933696252104]
  We introduce Iterative Reward-Guided Refinement (IterRef), a novel test-time scaling method tailored to discrete diffusion.<n>We formalize this process within a Multiple-Try Metropolis framework, proving convergence to the reward-aligned distribution.<n>IterRef achieves striking gains under low compute budgets, far surpassing prior state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-11-04T02:33:23Z)
• Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervals [48.14879329270912]
  Phased DMD is a multi-step distillation framework that bridges the idea of phase-wise distillation with Mixture-of-Experts.<n>Phased DMD is built upon two key ideas: progressive distribution matching and score matching within subintervals.<n> Experimental results demonstrate that Phased DMD preserves output diversity better than DMD while retaining key generative capabilities.
  arXiv  Detail & Related papers  (2025-10-31T17:55:10Z)
• Score Distillation of Flow Matching Models [67.86066177182046]
  We extend Score identity Distillation (SiD) to pretrained text-to-image flow-matching models.<n>SiD works out of the box across these models, in both data-free and data-aided settings.<n>This provides the first systematic evidence that score distillation applies broadly to text-to-image flow matching models.
  arXiv  Detail & Related papers  (2025-09-29T17:45:48Z)
• Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis [65.77083310980896]
  We propose Adrial Distribution Matching (ADM) to align latent predictions between real and fake score estimators for score distillation.<n>Our proposed method achieves superior one-step performance on SDXL compared to DMD2 while consuming less GPU time.<n>Additional experiments that apply multi-step ADM distillation on SD3-Medium, SD3.5-Large, and CogVideoX set a new benchmark towards efficient image and video synthesis.
  arXiv  Detail & Related papers  (2025-07-24T16:45:05Z)
• Broadening Target Distributions for Accelerated Diffusion Models via a Novel Analysis Approach [49.97755400231656]
  We show that a new accelerated DDPM sampler achieves accelerated performance for three broad distribution classes not considered before.<n>Our results show an improved dependency on the data dimension $d$ among accelerated DDPM type samplers.
  arXiv  Detail & Related papers  (2024-02-21T16:11:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.