Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization

• URL: http://arxiv.org/abs/2605.15980v1
• Date: Fri, 15 May 2026 14:13:39 GMT
• Title: Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization
• Abstract summary: Group Relative Policy Optimization is essential for aligning video diffusion models with human preferences.<n>Existing efficiency methods compromise optimization, exhibiting severe instability and failing to reach full trajectory performance.<n>We present Flash-GRPO, a single-step training framework that outperforms full trajectory training in alignment quality under low computational budgets.
• Score: 69.03519018775779
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Group Relative Policy Optimization has emerged as essential for aligning video diffusion models with human preferences, but faces a critical computational bottleneck: training a 14B parametered model typically demands hundreds of GPU days per experiment. Existing efficiency methods reduce costs through sliding window subsampling training timesteps, but fundamentally compromise optimization, exhibiting severe instability and failing to reach full trajectory performance. We present Flash-GRPO, a single-step training framework that outperforms full trajectory training in alignment quality under low computational budgets while substantially improving training efficiency. Flash-GRPO addresses two critical challenges: iso-temporal grouping eliminates timestep-confounded variance by enforcing prompt-wise temporal consistency, decoupling policy performance from timestep difficulty; temporal gradient rectification neutralizes the time-dependent scaling factor that causes vastly inconsistent gradient magnitudes across timesteps. Experiments on 1.3B to 14B parameter models validate Flash-GRPO's effectiveness, demonstrating substantial training acceleration with consistent stability and state-of-the-art alignment quality.

Related papers

• Scaling Reinforcement Learning for Diffusion Models via Velocity Matching [66.34541945134916]
  RewardBench fine-tuning is an important tool for adapting diffusion models to human preferences and task-specific objectives.<n>Unlike autoregressive models, diffusion models do not provide tractable likelihoods for generated samples.<n>We propose reward-based velocity matching (RVM), a simple trajectory-free update that acts directly on the velocity field.
  arXiv  Detail & Related papers  (2026-08-24T17:13:43Z)
• Explicit Critic Guidance for Aligning Diffusion Models [27.6704888666835]
  We propose a state-aligned latent actor-critic framework for diffusion post-training.<n>Our method consistently outperforms prior group-relative RL and actor-critic baselines on single-reward and multi-reward benchmarks.
  arXiv  Detail & Related papers  (2026-05-26T22:20:51Z)
• Diffusion-APO: Trajectory-Aware Direct Preference Alignment for Video Diffusion Transformers [12.948398661304184]
  Diffusion-APO is a trajectory-aware algorithm for aligning video diffusion models with human intent.<n>We introduce a unified and modular RLHF framework that integrates online ranking, half-online anchoring, offline refinement, and distillation-aware drift correction.<n>We demonstrate that Diffusion-APO consistently outperforms standard baselines in visual quality and instruction following.
  arXiv  Detail & Related papers  (2026-05-08T09:37:46Z)
• A Systematic Post-Train Framework for Video Generation [76.26555417456773]
  Large-scale video diffusion models have demonstrated impressive capabilities in generating high-resolution and semantically rich content.<n>A significant gap remains between their pretraining performance and real-world deployment requirements due to critical issues such as prompt sensitivity, temporal inconsistency, and prohibitive inference costs.<n>We propose a comprehensive post-training framework that systematically aligns pretrained models with user intentions through four synergistic stages.
  arXiv  Detail & Related papers  (2026-04-28T09:34:51Z)
• Adaptive Linear Path Model-Based Diffusion [52.84663832658799]
  We introduce Linear Path Model-Based Diffusion (LP-MBD), which replaces the variance-preserving schedule with a flow-matching linear probability path.<n>We also propose Adaptive LP-MBD (ALP-MBD), which leverages reinforcement learning to adjust diffusion steps and noise levels according to task complexity and environmental conditions.
  arXiv  Detail & Related papers  (2026-02-02T21:33:03Z)
• Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models [52.48582333951919]
  We propose a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates.<n>SAGE (Stability-Aware Gradient Efficiency) integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines.
  arXiv  Detail & Related papers  (2026-02-01T12:56:10Z)
• Hierarchical Schedule Optimization for Fast and Robust Diffusion Model Sampling [19.936149710230275]
  We present the Hierarchical-Schedule-r (HSO), a novel and efficient bi-level optimization framework.<n>HSO sets a new state-of-the-art for training-free sampling in the extremely low-NFE regime.<n>For instance, with an NFE of just 5, HSO achieves a remarkable FID of 11.94 on LAION-Aesthetics with Stable Diffusion v2.1.
  arXiv  Detail & Related papers  (2025-11-12T08:57:46Z)
• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs [53.749193998004166]
  Curriculum learning plays a crucial role in enhancing the training efficiency of large language models.<n>We propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead.
  arXiv  Detail & Related papers  (2025-10-01T15:41:27Z)
• DriftLite: Lightweight Drift Control for Inference-Time Scaling of Diffusion Models [22.823183347642132]
  We study inference-time scaling for diffusion models, where the goal is to adapt a pre-trained model to new target distributions without retraining.<n>We introduce DriftLite, a lightweight, training-free particle-based approach that steers the inference dynamics on the fly with provably optimal stability.
  arXiv  Detail & Related papers  (2025-09-25T22:21:59Z)
• A-FloPS: Accelerating Diffusion Sampling with Adaptive Flow Path Sampler [21.134678093577193]
  A-FloPS is a principled, training-free framework for flow-based generative models.<n>We show that A-FloPS consistently outperforms state-of-the-art training-free samplers in both sample quality and efficiency.<n>With as few as $5$ function evaluations, A-FloPS achieves substantially lower FID and generates sharper, more coherent images.
  arXiv  Detail & Related papers  (2025-08-22T13:28:16Z)
• Fast and Stable Diffusion Planning through Variational Adaptive Weighting [3.745003761050674]
  Diffusion models have recently shown promise in offline RL.<n>These methods often suffer from high training costs and slow convergence.<n>We introduce a closed-form approximation method for its online estimation under the flow-based generative modeling framework.<n> Experimental results on Maze2D and Kitchen tasks show that our method achieves competitive performance with up to 10 times fewer training steps.
  arXiv  Detail & Related papers  (2025-06-20T02:12:04Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.