Video Models Can Reason with Verifiable Rewards

• URL: http://arxiv.org/abs/2605.15458v1
• Date: Thu, 14 May 2026 22:40:56 GMT
• Title: Video Models Can Reason with Verifiable Rewards
• Abstract summary: We introduce VideoRLVR, a recipe for optimizing video diffusion models with rule-based feedback.<n>VideoRLVR formulates video reasoning as the generation of verifiable visual trajectories.<n>We evaluate VideoRLVR on Maze, FlowFree, and Sokoban, three procedurally generated domains with objective success criteria.
• Score: 31.381840584972675
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video diffusion models have made rapid progress in perceptual realism and temporal coherence, but they remain primarily optimized for plausible generation rather than verifiable reasoning. This limitation is especially pronounced in tasks where generated videos must satisfy explicit spatial, temporal, or logical constraints. Inspired by the role of reinforcement learning with verifiable rewards (RLVR) in reasoning-oriented language models, we introduce VideoRLVR, a practical recipe for optimizing video diffusion models with rule-based feedback. VideoRLVR formulates video reasoning as the generation of verifiable visual trajectories and consists of an SDE-GRPO optimization backbone, dense decomposed rewards, and an Early-Step Focus strategy for efficient training. The Early-Step Focus strategy restricts policy optimization to the early denoising phase, reducing training latency by about 40% while preserving performance. We evaluate VideoRLVR on Maze, FlowFree, and Sokoban, three procedurally generated domains with objective success criteria. Across these tasks, VideoRLVR consistently improves over supervised fine-tuning baselines, with dense decomposed rewards proving especially important in low-success-rate settings. Our RL-optimized model also outperforms the evaluated proprietary and open-source video generation models on these verifiable reasoning benchmarks and out-of-domain benchmarks. These results suggest that verifiable RL can move video models beyond perceptual imitation toward more reliable rule-consistent visual reasoning.

Related papers

• From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models [42.398664944109676]
  We introduce VWG-Bench (Video World Generalist Benchmark), a comprehensive benchmark spanning 9 reasoning dimensions and 38 fine-grained tasks.<n>We propose Vid-PRE (Video Reason Prompter and Enhancer), a model-agnostic prompt rewriter that offloads the cognitive burden of reasoning to a dedicated VLM.
  arXiv  Detail & Related papers  (2026-09-10T08:40:58Z)
• CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL [56.969946199335716]
  We introduce a compositional constraint-based reward model for post-training embodied video generation models.<n>We propose CreFlow, a novel online RL framework with two key designs.
  arXiv  Detail & Related papers  (2026-05-14T02:18:58Z)
• Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity [57.83511884904928]
  We explore methods to accelerate training for autoregressive video generation models through empirical analyses.<n>Our results reveal that while training on fewer video frames significantly reduces training time, it also exacerbates error accumulation and introduces inconsistencies in the generated videos.<n>Inspired by Lipschitz continuity, we propose a Representation Continuity (ReCo) strategy to improve the consistency of generated videos.
  arXiv  Detail & Related papers  (2026-04-08T09:43:03Z)
• Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models [58.3184497327891]
  Distilled autoregressive (AR) video models enable efficient streaming generation but frequently misalign with human visual preferences.<n>We present Astrolabe, an efficient online reinforcement learning framework tailored for distilled AR models.
  arXiv  Detail & Related papers  (2026-03-17T18:32:18Z)
• Video Generation Models Are Good Latent Reward Models [35.261036369253524]
  Reward feedback learning (ReFL) has proven effective for aligning image generation with human preferences.<n>Existing video reward models rely on vision-language models designed for pixel-space inputs.<n>Process Reward Feedback Learning(PRFL) is a framework that conducts preference optimization entirely in latent space.
  arXiv  Detail & Related papers  (2025-11-26T16:14:18Z)
• SimpleGVR: A Simple Baseline for Latent-Cascaded Video Super-Resolution [46.311223206965934]
  We study key design principles for latter cascaded video super-resolution models, which are underexplored currently.<n>First, we propose two strategies to generate training pairs that better mimic the output characteristics of the base model, ensuring alignment between the VSR model and its upstream generator.<n>Second, we provide critical insights into VSR model behavior through systematic analysis of (1) timestep sampling strategies, (2) noise augmentation effects on low-resolution (LR) inputs.
  arXiv  Detail & Related papers  (2025-06-24T17:57:26Z)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection [35.268183415853976]
  We provide an explanation of the RL training process through empirical analysis and rigorous theoretical modeling.<n>We develop a theoretical framework to understand the training dynamics of RL with two typical rewards: reward (RLVR) and model's internal feedback (RLIF)
  arXiv  Detail & Related papers  (2025-06-05T07:17:04Z)
• Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency [56.475612147721264]
  We propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals.<n>We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA.<n>Results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs.
  arXiv  Detail & Related papers  (2025-06-02T17:28:26Z)
• ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning [68.76048244253582]
  We introduce ViaRL, the first framework to leverage rule-based reinforcement learning (RL) for optimizing frame selection in video understanding.<n>ViaRL utilizes the answer accuracy of a downstream model as a reward signal to train a frame selector through trial-and-error.<n>ViaRL consistently delivers superior temporal grounding performance and robust generalization across diverse video understanding tasks.
  arXiv  Detail & Related papers  (2025-05-21T12:29:40Z)
• Improving Video Generation with Human Feedback [105.81833319891537]
  We develop a systematic pipeline that harnesses human feedback to mitigate video generation problems.<n>We introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy.
  arXiv  Detail & Related papers  (2025-01-23T18:55:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.