Manifold-Aware Exploration for Reinforcement Learning in Video Generation

• URL: http://arxiv.org/abs/2603.21872v1
• Date: Mon, 23 Mar 2026 12:04:38 GMT
• Title: Manifold-Aware Exploration for Reinforcement Learning in Video Generation
• Abstract summary: Group Relative Policy Optimization (GRPO) methods for video generation like FlowGRPO remain far less reliable than their counterparts for language models and images.<n>This gap arises because video generation has a complex solution space, and the ODE-to-SDE conversion used for exploration can inject excess noise.<n>We propose SAGE-GRPO (Stable Alignment via Exploration), which applies constraints at both micro and macro levels.
• Score: 62.48300523142513
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Group Relative Policy Optimization (GRPO) methods for video generation like FlowGRPO remain far less reliable than their counterparts for language models and images. This gap arises because video generation has a complex solution space, and the ODE-to-SDE conversion used for exploration can inject excess noise, lowering rollout quality and making reward estimates less reliable, which destabilizes post-training alignment. To address this problem, we view the pre-trained model as defining a valid video data manifold and formulate the core problem as constraining exploration within the vicinity of this manifold, ensuring that rollout quality is preserved and reward estimates remain reliable. We propose SAGE-GRPO (Stable Alignment via Exploration), which applies constraints at both micro and macro levels. At the micro level, we derive a precise manifold-aware SDE with a logarithmic curvature correction and introduce a gradient norm equalizer to stabilize sampling and updates across timesteps. At the macro level, we use a dual trust region with a periodic moving anchor and stepwise constraints so that the trust region tracks checkpoints that are closer to the manifold and limits long-horizon drift. We evaluate SAGE-GRPO on HunyuanVideo1.5 using the original VideoAlign as the reward model and observe consistent gains over previous methods in VQ, MQ, TA, and visual metrics (CLIPScore, PickScore), demonstrating superior performance in both reward maximization and overall video quality. The code and visual gallery are available at https://dungeonmassster.github.io/SAGE-GRPO-Page/.

Related papers

• LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction [59.32834240526981]
  Flow-based generative models are typically sampled by solving a deterministic ordinary differential equation (ODE)<n>Existing GRPO methods for flow models replace the Euler-time ODE with aotropic differential equation (SDE) during training.<n>We introduce LC-GRPO, a flow-based GRPO framework with Langevin correction targeting the marginal distribution at the resulting timestep.
  arXiv  Detail & Related papers  (2026-08-06T04:47:57Z)
• Self Gradient Forcing: Native Long Video Extrapolation [60.00513917489133]
  Self Gradient Forcing restores historical context-gradient gap.<n>Code and models will be released to advance research on autoregressive video generation.
  arXiv  Detail & Related papers  (2026-07-22T16:59:23Z)
• RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO [53.38929612273108]
  Causal autoregressive video diffusion models support real-time streaming generation by extrapolating future chunks from previously generated content.<n>We introduce the Real-time Autoregressive Video Extrapolation Network (RAVEN), a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints.
  arXiv  Detail & Related papers  (2026-05-14T17:59:30Z)
• Salt: Self-Consistent Distribution Matching with Cache-Aware Training for Fast Video Generation [27.698320788533405]
  Trajectory-style consistency distillation (DMD) can recover sharp, mode-seeking samples, but its local training signals do not explicitly regularize how denoising updates compose across timesteps.<n>We propose Self-Consistent Distribution Matching Distillation (SC-DMD), which explicitly regularizes the endpoint-consistent composition of consecutive denoising updates.
  arXiv  Detail & Related papers  (2026-04-03T15:43:21Z)
• Free-Lunch Long Video Generation via Layer-Adaptive O.O.D Correction [12.736570247861309]
  We propose FreeLOC, a training-free, layer-adaptive framework for long-video inference.<n>Our approach significantly outperforms existing training-free methods, achieving state-of-the-art results in both temporal consistency and visual quality.
  arXiv  Detail & Related papers  (2026-03-26T09:12:14Z)
• VIGOR: VIdeo Geometry-Oriented Reward for Temporal Generative Alignment [15.619170225414571]
  Video diffusion models lack explicit geometric supervision during training, leading to inconsistency artifacts.<n>We propose a geometry-based reward model that leverages pretrained geometric foundation models to evaluate multi-view consistency.<n>Our approach conducts error computation in a pointwise fashion, yielding a more physically grounded and robust error metric.
  arXiv  Detail & Related papers  (2026-03-17T09:04:10Z)
• VideoHEDGE: Entropy-Based Hallucination Detection for Video-VLMs via Semantic Clustering and Spatiotemporal Perturbations [4.509454543418357]
  Hallucinations in video-capable vision models (VideoVLMs) remain frequent and high-confidence.<n>We introduce VideoHEDGE, a modular framework for hallucination detection in question answering.
  arXiv  Detail & Related papers  (2026-01-13T13:42:05Z)
• Mind the Generative Details: Direct Localized Detail Preference Optimization for Video Diffusion Models [65.16788152626499]
  LocalDPO builds a novel framework for aligning video diffusion models with human preferences.<n>We show that LocalDPO consistently improves video fidelity, temporal coherence and human preference scores over other post-training approaches.
  arXiv  Detail & Related papers  (2026-01-07T16:32:17Z)
• CAMP-VQA: Caption-Embedded Multimodal Perception for No-Reference Quality Assessment of Compressed Video [9.172799792564009]
  We propose CAMP-VQA, a novel NR-VQA framework that exploits the semantic understanding capabilities of large models.<n>Our approach introduces a quality-aware video metadata mechanism that integrates key fragments extracted from inter-frame variations.<n>Our model consistently outperforms existing NR-VQA methods, achieving improved accuracy without the need for costly manual fine-grained annotations.
  arXiv  Detail & Related papers  (2025-11-10T16:37:47Z)
• HyPCV-Former: Hyperbolic Spatio-Temporal Transformer for 3D Point Cloud Video Anomaly Detection [1.475698751142657]
  HyV-Former achieves state-of-the-art anomaly detection across multiple anomaly categories, with a 7% improvement on the TIMo dataset and a 5.6% gain on the DAD dataset.
  arXiv  Detail & Related papers  (2025-08-01T09:50:20Z)
• Transform-Equivariant Consistency Learning for Temporal Sentence Grounding [66.10949751429781]
  We introduce a novel Equivariant Consistency Regulation Learning framework to learn more discriminative representations for each video. Our motivation comes from that the temporal boundary of the query-guided activity should be consistently predicted. In particular, we devise a self-supervised consistency loss module to enhance the completeness and smoothness of the augmented video.
  arXiv  Detail & Related papers  (2023-05-06T19:29:28Z)
• Flow-Guided Sparse Transformer for Video Deblurring [124.11022871999423]
  FlowGuided Sparse Transformer (F GST) is a framework for video deblurring. FGSW-MSA enjoys the guidance of the estimated optical flow to globally sample spatially sparse elements corresponding to the same scene patch in neighboring frames. Our proposed F GST outperforms state-of-the-art patches on both DVD and GOPRO datasets and even yields more visually pleasing results in real video deblurring.
  arXiv  Detail & Related papers  (2022-01-06T02:05:32Z)
• Robust Unsupervised Video Anomaly Detection by Multi-Path Frame Prediction [61.17654438176999]
  We propose a novel and robust unsupervised video anomaly detection method by frame prediction with proper design. Our proposed method obtains the frame-level AUROC score of 88.3% on the CUHK Avenue dataset.
  arXiv  Detail & Related papers  (2020-11-05T11:34:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.