Video Generation Models Are Good Latent Reward Models

• URL: http://arxiv.org/abs/2511.21541v1
• Date: Wed, 26 Nov 2025 16:14:18 GMT
• Title: Video Generation Models Are Good Latent Reward Models
• Abstract summary: Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences.<n>Existing video reward models rely on vision-language models designed for pixel-space inputs.<n>Process Reward Feedback Learning(PRFL) is a framework that conducts preference optimization entirely in latent space.
• Score: 35.261036369253524
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences. However, its extension to video generation faces significant challenges. Existing video reward models rely on vision-language models designed for pixel-space inputs, confining ReFL optimization to near-complete denoising steps after computationally expensive VAE decoding. This pixel-space approach incurs substantial memory overhead and increased training time, and its late-stage optimization lacks early-stage supervision, refining only visual quality rather than fundamental motion dynamics and structural coherence. In this work, we show that pre-trained video generation models are naturally suited for reward modeling in the noisy latent space, as they are explicitly designed to process noisy latent representations at arbitrary timesteps and inherently preserve temporal information through their sequential modeling capabilities. Accordingly, we propose Process Reward Feedback Learning~(PRFL), a framework that conducts preference optimization entirely in latent space, enabling efficient gradient backpropagation throughout the full denoising chain without VAE decoding. Extensive experiments demonstrate that PRFL significantly improves alignment with human preferences, while achieving substantial reductions in memory consumption and training time compared to RGB ReFL.

Related papers

• Spectral Progressive Diffusion for Efficient Image and Video Generation [38.48147404244147]
  We propose a general framework that progressively grows resolution along the denoising trajectory of pretrained diffusion models.<n>Our framework supports training-free acceleration and a novel fine-tuning recipe that further improves efficiency and quality.
  arXiv  Detail & Related papers  (2026-05-18T17:55:39Z)
• Video Models Can Reason with Verifiable Rewards [31.381840584972675]
  We introduce VideoRLVR, a recipe for optimizing video diffusion models with rule-based feedback.<n>VideoRLVR formulates video reasoning as the generation of verifiable visual trajectories.<n>We evaluate VideoRLVR on Maze, FlowFree, and Sokoban, three procedurally generated domains with objective success criteria.
  arXiv  Detail & Related papers  (2026-05-14T22:40:56Z)
• Video Generation with Predictive Latents [50.3100375593545]
  Video Autoencoder (Eational) enables latent video generative modeling by mapping the visual world into compact latent spaces.<n>How to enhance the diffusability of video latents remains a critical and unresolved challenge.<n>We introduce a simple and effective predictive reconstruction objective that unifies predictive learning with video reconstruction.
  arXiv  Detail & Related papers  (2026-05-04T01:30:04Z)
• Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity [57.83511884904928]
  We explore methods to accelerate training for autoregressive video generation models through empirical analyses.<n>Our results reveal that while training on fewer video frames significantly reduces training time, it also exacerbates error accumulation and introduces inconsistencies in the generated videos.<n>Inspired by Lipschitz continuity, we propose a Representation Continuity (ReCo) strategy to improve the consistency of generated videos.
  arXiv  Detail & Related papers  (2026-04-08T09:43:03Z)
• Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models [58.3184497327891]
  Distilled autoregressive (AR) video models enable efficient streaming generation but frequently misalign with human visual preferences.<n>We present Astrolabe, an efficient online reinforcement learning framework tailored for distilled AR models.
  arXiv  Detail & Related papers  (2026-03-17T18:32:18Z)
• Learning from Next-Frame Prediction: Autoregressive Video Modeling Encodes Effective Representations [53.91818843831925]
  We propose NExT-Vid, a novel autoregressive visual generative pretraining framework.<n>We introduce a context-isolated autoregressive predictor to decouple semantic representation from target decoding.<n>Through context-isolated flow-matching pretraining, our approach achieves strong representations.
  arXiv  Detail & Related papers  (2025-12-24T07:07:08Z)
• Training-Free Motion Customization for Distilled Video Generators with Adaptive Test-Time Distillation [53.877572078307935]
  Distilled video generation models offer fast and efficient but struggle with motion customization when guided by reference videos.<n>We propose MotionEcho, a training-free test-time distillation framework that enables motion customization by leveraging diffusion teacher forcing.
  arXiv  Detail & Related papers  (2025-06-24T06:20:15Z)
• Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization [46.50233461744791]
  Preference optimization for diffusion models aims to align them with human preferences for images.<n>We show that pre-trained diffusion models are naturally suited for step-level reward modeling in the noisy latent space.<n>We introduce Latent Preference Optimization (LPO), a step-level preference optimization method conducted directly in the noisy latent space.
  arXiv  Detail & Related papers  (2025-02-03T04:51:28Z)
• Autoregressive Video Generation without Vector Quantization [90.87907377618747]
  We reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction.<n>With the proposed approach, we train a novel video autoregressive model without vector quantization, termed NOVA.<n>Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity.
  arXiv  Detail & Related papers  (2024-12-18T18:59:53Z)
• RL for Consistency Models: Faster Reward Guided Text-to-Image Generation [15.238373471473645]
  We propose a framework for fine-tuning consistency models viaReinforcement Learning (RL) Our framework, called Reinforcement Learning for Consistency Model (RLCM), frames the iterative inference process of a consistency model as an RL procedure. Comparing to RL finetuned diffusion models, RLCM trains significantly faster, improves the quality of the generation measured under the reward objectives, and speeds up the inference procedure by generating high quality images with as few as two inference steps.
  arXiv  Detail & Related papers  (2024-03-25T15:40:22Z)
• Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptation [112.08287900261898]
  This paper proposes a novel self-cascade diffusion model for rapid adaptation to higher-resolution image and video generation. Our approach achieves a 5X training speed-up and requires only an additional 0.002M tuning parameters. Experiments demonstrate that our approach can quickly adapt to higher resolution image and video synthesis by fine-tuning for just 10k steps, with virtually no additional inference time.
  arXiv  Detail & Related papers  (2024-02-16T07:48:35Z)
• Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction [61.16125290912494]
  $textEVL_textGen$ is a framework designed for the pre-training of visually conditioned language generation models. We show that our approach accelerates the training of vision-language models by a factor of 5 without a noticeable impact on overall performance.
  arXiv  Detail & Related papers  (2023-10-05T03:40:06Z)
• Leaping Into Memories: Space-Time Deep Feature Synthesis [93.10032043225362]
  We propose LEAPS, an architecture-independent method for synthesizing videos from internal models. We quantitatively and qualitatively evaluate the applicability of LEAPS by inverting a range of architectures convolutional attention-based on Kinetics-400.
  arXiv  Detail & Related papers  (2023-03-17T12:55:22Z)
• Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modeling [28.530765643908083]
  We decouple spatial-temporal modeling and integrate an image- and a video-language to learn fine-grained visual understanding. We propose a novel pre-training objective, Temporal Referring Modeling, which requires the model to identify temporal positions of events in video sequences. Our model outperforms previous work pre-trained on orders of magnitude larger datasets.
  arXiv  Detail & Related papers  (2022-10-08T07:03:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.