Growing with the Generator: Self-paced GRPO for Video Generation

• URL: http://arxiv.org/abs/2511.19356v1
• Date: Mon, 24 Nov 2025 17:56:03 GMT
• Title: Growing with the Generator: Self-paced GRPO for Video Generation
• Abstract summary: Group Relative Policy Optimization has emerged as a powerful reinforcement learning paradigm for post-training video generation models.<n>We propose Self-Paced GRPO, a competence-aware GRPO framework in which reward feedback co-evolves with the generator.<n>Our method introduces a progressive reward mechanism that shifts its emphasis from coarse visual fidelity to temporal coherence and fine-grained text-video semantic alignment as generation quality increases.
• Score: 45.5073437581357
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Group Relative Policy Optimization (GRPO) has emerged as a powerful reinforcement learning paradigm for post-training video generation models. However, existing GRPO pipelines rely on static, fixed-capacity reward models whose evaluation behavior is frozen during training. Such rigid rewards introduce distributional bias, saturate quickly as the generator improves, and ultimately limit the stability and effectiveness of reinforcement-based alignment. We propose Self-Paced GRPO, a competence-aware GRPO framework in which reward feedback co-evolves with the generator. Our method introduces a progressive reward mechanism that automatically shifts its emphasis from coarse visual fidelity to temporal coherence and fine-grained text-video semantic alignment as generation quality increases. This self-paced curriculum alleviates reward-policy mismatch, mitigates reward exploitation, and yields more stable optimization. Experiments on VBench across multiple video generation backbones demonstrate consistent improvements in both visual quality and semantic alignment over GRPO baselines with static rewards, validating the effectiveness and generality of Self-Paced GRPO.

Related papers

• Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training [8.16921896191724]
  Autoregressive text-to-image (T2I) generation has recently advanced rapidly, yet aligning generated images with human preferences remains challenging.<n>We study this overlooked factor within a unified f-divergence framework, encompassing forward KL, reverse KL, and JS divergence.<n>Our systematic theoretical analysis reveals that different divergences reshape token-level updates in distinct ways.
  arXiv  Detail & Related papers  (2026-06-19T14:50:22Z)
• AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward [52.7128440950672]
  We propose AlphaGRPO to enhance multimodal generation capabilities without an additional cold-start stage.<n>Our approach unlocks the model's intrinsic potential to perform advanced reasoning tasks.<n>To address the challenge of providing stable supervision for real-world multimodal generation, we introduce the Decompositional Verifiable Reward (DVReward)
  arXiv  Detail & Related papers  (2026-05-12T17:59:47Z)
• OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models [14.396100082949005]
  We present OP-GRPO, the first Off-Policy GRPO framework tailored for flow-matching models.<n>We actively select high-quality trajectories and adaptively incorporate them into a replay buffer for reuse in subsequent training iterations.
  arXiv  Detail & Related papers  (2026-04-05T15:00:29Z)
• UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation [51.41441081823758]
  Unified models capable of interleaved generation have emerged as a promising paradigm.<n>We propose a unified reinforcement learning framework tailored for interleaved generation.<n>Our experiments demonstrate that this unified training recipe significantly enhances image generation quality through reasoning.
  arXiv  Detail & Related papers  (2026-03-24T17:59:17Z)
• AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization [47.39648302477585]
  Streaming autoregressive (AR) video generators combined with few-step distillation achieve low-latency, high-quality synthesis.<n>We propose AR-CoPO, a framework that adapts the Neighbor GRPO contrastive perspective to streaming AR generation.
  arXiv  Detail & Related papers  (2026-03-18T08:07:01Z)
• iGRPO: Self-Feedback-Driven LLM Reasoning [88.83313431248473]
  Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions.<n>We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts.<n>Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models.
  arXiv  Detail & Related papers  (2026-02-09T18:45:11Z)
• Adaptive-Boundary-Clipping GRPO: Ensuring Bounded Ratios for Stable and Generalizable Training [7.404779700134294]
  Adaptive-Boundary-Clipping GRPO (ABC-GRPO) is an asymmetric and adaptive refinement of the original GRPO framework.<n>ABC-GRPO achieves superior performance over standard GRPO on mathematical reasoning tasks.
  arXiv  Detail & Related papers  (2026-01-07T13:04:52Z)
• Generative Actor Critic [74.04971271003869]
  Generative Actor Critic (GAC) is a novel framework that decouples sequential decision-making by reframing textitpolicy evaluation as learning a generative model of the joint distribution over trajectories and returns.<n>Experiments on Gym-MuJoCo and Maze2D benchmarks demonstrate GAC's strong offline performance and significantly enhanced offline-to-online improvement compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-12-25T06:31:11Z)
• Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment [61.80228667422234]
  VGPO redefines value estimation across both temporal and group dimensions.<n>It transforms the sparse terminal reward into dense, process-aware value estimates.<n>It replaces standard group normalization with a novel process enhanced by absolute values to maintain a stable optimization signal.
  arXiv  Detail & Related papers  (2025-12-13T16:31:26Z)
• MURPHY: Multi-Turn GRPO for Self Correcting Code Generation [55.66642560374686]
  Murphy is a multi-turn reflective optimization framework that extends GRPO by incorporating iterative self-correction during training.<n>We show that Murphy consistently improves performance, achieving up to a 8% relative gain in pass@1 over GRPO, on similar compute budgets.
  arXiv  Detail & Related papers  (2025-11-11T05:03:22Z)
• GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping [63.33669214116784]
  GRPO-Guard is a simple yet effective enhancement to existing GRPO frameworks.<n>It restores a balanced and step-consistent importance ratio, ensuring that PPO clipping properly constrains harmful updates.<n>It substantially mitigates implicit over-optimization without relying on heavy KL regularization.
  arXiv  Detail & Related papers  (2025-10-25T14:51:17Z)
• BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models [57.304411396229035]
  We present BranchGRPO, a method that restructures the rollout process into a branching tree.<n>On HPDv2.1 image alignment, BranchGRPO improves alignment scores by up to textbf16% over DanceGRPO.<n>A hybrid variant, BranchGRPO-Mix, further accelerates training to 4.7x faster than DanceGRPO without degrading alignment.
  arXiv  Detail & Related papers  (2025-09-07T12:53:06Z)
• TempFlow-GRPO: When Timing Matters for GRPO in Flow Models [22.023027865557637]
  We introduce a principled GRPO framework that captures and exploits the temporal structure inherent in flow-based generation.<n>New innovations endow the model with temporally-aware optimization that respects the underlying generative dynamics.
  arXiv  Detail & Related papers  (2025-08-06T11:10:39Z)
• Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimization [0.0]
  Hybrid Group Relative Policy Optimization (Hybrid GRPO) is a reinforcement learning framework.<n>It incorporates empirical multi-sample action evaluation while preserving the stability of value function-based learning.<n>By integrating structured empirical sampling with reinforcement learning stability mechanisms, Hybrid GRPO has potential applications in autonomous robotics, financial modeling, and AI-driven control systems.
  arXiv  Detail & Related papers  (2025-01-30T21:04:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.