Scaling Reinforcement Learning for Diffusion Models via Velocity Matching
Abstract Overview
This paper argues that reward fine-tuning for diffusion models does not need likelihood-based policy-gradient machinery and instead can be formulated directly in the model's native velocity representation. The authors propose reward-based velocity matching (RVM), a trajectory-free objective that applies reward-weighted regression on a single noised state, with an optional anchor term to control drift from a reference velocity field. They show that recent velocity-based fine-tuning methods, including RAM and DiffusionNFT, can be written as special cases of the same update structure. Experiments across text-to-image, text-to-video, and image-to-video models compare RVM with trajectory-based and other velocity-based baselines in terms of quality and training cost. The evidence also emphasizes that reward design, especially for motion in video generation, is more consequential than the precise algebraic form of the velocity loss.
Novelty
The main novelty is a simplified reward fine-tuning formulation for diffusion models that removes policy-ratio estimation and denoising-trajectory storage, acting directly on velocity fields. The paper also contributes a unifying view that places RAM and DiffusionNFT within the same RVM framework, and introduces a dynamic-tracking reward to address the tendency of standard video rewards to prefer static outputs.
Results
Across video tasks, velocity-matching methods consistently outperform trajectory-based fine-tuning while using much less training compute; for Wan2.1-T2V-1.3B, RVM reaches the highest reported VBench Overall score (84.13) and dynamic degree (75.00), and for SkyReels-I2V it improves Overall to 86.27 while FlowGRPO falls below the base model. On SD3.5-M for OCR-oriented text-to-image fine-tuning, RVM achieves the best or tied-best results on most reported metrics. The reported cost for a full Wan2.1-T2V-1.3B run is 525 GPU-hours for RVM versus 1,159 for FlowGRPO and 6,171 for DanceGRPO/TaRoS.
Key Points
- RVM fine-tunes diffusion models with reward-weighted velocity regression on single noised samples, avoiding likelihood-ratio estimation and full trajectory storage.
- RAM and DiffusionNFT are shown to be special cases of the same anchored velocity-matching update, suggesting that loss-form differences are less important than reward and anchor choices.
- For video generation, adding the proposed dynamic-tracking reward markedly improves motion-related evaluation, highlighting reward specification as a key determinant of downstream quality.