Scaling Reinforcement Learning for Diffusion Models via Velocity Matching

• URL: http://arxiv.org/abs/2608.23664v1
• Date: Mon, 24 Aug 2026 17:13:43 GMT
• Title: Scaling Reinforcement Learning for Diffusion Models via Velocity Matching
• Abstract summary: RewardBench fine-tuning is an important tool for adapting diffusion models to human preferences and task-specific objectives.<n>Unlike autoregressive models, diffusion models do not provide tractable likelihoods for generated samples.<n>We propose reward-based velocity matching (RVM), a simple trajectory-free update that acts directly on the velocity field.
• Score: 66.34541945134916
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reward fine-tuning is becoming an important tool for adapting diffusion models to human preferences and task-specific objectives, but existing methods largely inherit policy-gradient machinery from large language models. Unlike autoregressive models, diffusion models do not provide tractable likelihoods for generated samples. As a result, current approaches either construct trajectory likelihoods from stochastic denoising transitions or approximate endpoint likelihoods with evidence lower bound, introducing additional computation and algorithmic complexity. We demonstrate that this likelihood-based machinery is not necessary for effective diffusion reward fine-tuning. We propose reward-based velocity matching (RVM), a simple trajectory-free update that acts directly on the velocity field. RVM reinforces directions associated with high-reward generations, suppresses those with low reward, and involves an optional anchor term controlling drift from a reference velocity. Notably, it provides a general framework that recovers recent fine-tuning methods, including RAM and DiffusionNFT, as special cases. Across various large-scale diffusion models reward fine-tuning tasks, RVM is competitive with or outperforms trajectory-based policy-gradient methods under substantially reduced training cost. We further find that, once the velocity update is simplified, the particular loss variant matters less than reward and anchor design. For video generation, standard preference rewards can favor visually clean but nearly static outputs; introducing a new dynamic-tracking reward that substantially improve motions while improving overall VBench performance. These results suggest that scalable reward fine-tuning for diffusion models is better posed in the native velocity representation than as likelihood-based policy optimization.

Related papers

• Perceptual Flow Matching for Few-Step Generative Modeling [64.26537634881244]
  Perceptual Flow Matching is a framework for few-step generation in flow-matching models.<n>We show that PFM produces high-quality few-step generators when supervised in an appropriate representation space.
  arXiv  Detail & Related papers  (2026-07-03T17:55:01Z)
• TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment [52.570581883709345]
  We propose Trajectory Matching Policy Balance Optimization (TMPO), which replaces reward with human-level reward distribution matching.<n>TMPO improves generative diversity over state-of-the-art methods by 9.1%, and achieves competitive performance in all downstream and efficiency metrics.<n>To further reduce multi-trajectory training time on large-scale flow- prefixes, TMPO incorporates Dynamic Tree Sampling models, where trajectories share denoising and branch at dynamically scheduled steps.
  arXiv  Detail & Related papers  (2026-05-09T04:41:02Z)
• Diffusion-APO: Trajectory-Aware Direct Preference Alignment for Video Diffusion Transformers [12.948398661304184]
  Diffusion-APO is a trajectory-aware algorithm for aligning video diffusion models with human intent.<n>We introduce a unified and modular RLHF framework that integrates online ranking, half-online anchoring, offline refinement, and distillation-aware drift correction.<n>We demonstrate that Diffusion-APO consistently outperforms standard baselines in visual quality and instruction following.
  arXiv  Detail & Related papers  (2026-05-08T09:37:46Z)
• Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models [54.597200388738656]
  Reward-based fine-tuning aims to steer a pretrained diffusion or flow-based generative model toward higher-reward samples.<n>We show that many can be written under a common framework, which we call reward score matching (RSM)
  arXiv  Detail & Related papers  (2026-04-19T12:47:52Z)
• Diffusion Reinforcement Learning via Centered Reward Distillation [35.979608265594685]
  We present textbf Reward Distillation (CRD), a diffusion RL framework derived from KL-regularized reward models built on forward-process fine-tuning.<n>To enable reliable text-to-image fine-tuning, we introduce techniques that explicitly control distribution drift.<n>Experiments on text-to-image post-training with textttGenEval and textttOCR rewards show that competitive SOTA reward optimization results with fast convergence and reduced reward hacking preference.
  arXiv  Detail & Related papers  (2026-03-14T21:29:33Z)
• FideDiff: Efficient Diffusion Model for High-Fidelity Image Motion Deblurring [33.809728459395785]
  We introduce FideDiff, a novel single-step diffusion model designed for high-fidelity deblurring.<n>We reformulate motion deblurring as a diffusion-like process where each timestep represents a progressively blurred image.<n>By reconstructing training data with matched blur trajectories, the model learns temporal consistency, enabling accurate one-step deblurring.
  arXiv  Detail & Related papers  (2025-10-02T03:44:45Z)
• A-FloPS: Accelerating Diffusion Sampling with Adaptive Flow Path Sampler [21.134678093577193]
  A-FloPS is a principled, training-free framework for flow-based generative models.<n>We show that A-FloPS consistently outperforms state-of-the-art training-free samplers in both sample quality and efficiency.<n>With as few as $5$ function evaluations, A-FloPS achieves substantially lower FID and generates sharper, more coherent images.
  arXiv  Detail & Related papers  (2025-08-22T13:28:16Z)
• One-Step Diffusion Model for Image Motion-Deblurring [85.76149042561507]
  We propose a one-step diffusion model for deblurring (OSDD), a novel framework that reduces the denoising process to a single step.<n>To tackle fidelity loss in diffusion models, we introduce an enhanced variational autoencoder (eVAE), which improves structural restoration.<n>Our method achieves strong performance on both full and no-reference metrics.
  arXiv  Detail & Related papers  (2025-03-09T09:39:57Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.