Wan-R1: Verifiable-Reinforcement Learning for Video Reasoning

• URL: http://arxiv.org/abs/2603.27866v1
• Date: Sun, 29 Mar 2026 20:53:00 GMT
• Title: Wan-R1: Verifiable-Reinforcement Learning for Video Reasoning
• Abstract summary: Video generation models produce visually coherent content but struggle with tasks requiring spatial reasoning and multi-step planning.<n>We design verifiable reward functions grounded in objective task metrics for maze-solving and robotic navigation tasks.<n>Experiments show that RL fine-tuning with verifiable rewards improves generalization.<n>Our systematic reward analysis reveals that verifiable rewards are critical for stable training, while multimodal reward models could lead to degenerate solutions.
• Score: 34.259169810657895
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video generation models produce visually coherent content but struggle with tasks requiring spatial reasoning and multi-step planning. Reinforcement learning (RL) offers a path to improve generalization, but its effectiveness in video reasoning hinges on reward design -- a challenge that has received little systematic study. We investigate this problem by adapting Group Relative Policy Optimization (GRPO) to flow-based video models and training them on maze-solving and robotic navigation tasks. We first show that multimodal reward models fail catastrophically in this setting. To address this, we design verifiable reward functions grounded in objective task metrics. For structured game environments, we introduce a multi-component trajectory reward. For robotic navigation, we propose an embedding-level verifiable reward. Our experiments show that RL fine-tuning with verifiable rewards improves generalization. For example, on complex 3D mazes, our model improves exact match accuracy by 29.1\% over the SFT baseline, and on trap-avoidance tasks by 51.4\%. Our systematic reward analysis reveals that verifiable rewards are critical for stable training, while multimodal reward models could lead to degenerate solutions. These findings establish verifiable reward design as a key enabler for robust video reasoning. Code will be publicly available.

Related papers

• RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents [24.215311212134882]
  We propose Reward-Swap Policy Optimization (RSPO) to leverage the rich information from dense process rewards to facilitate training with outcome rewards.<n>RSPO ensures the diversity of sampled trajectories while guaranteeing consistency between the optimization objective and the true outcome rewards.
  arXiv  Detail & Related papers  (2026-07-06T06:32:39Z)
• Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation [46.38008143057758]
  Large Language Models (LLMs) demonstrate transformative potential, yet their reasoning remains inconsistent and unreliable.<n>This work argues that reward modeling is not merely an implementation detail but a central architect of reasoning alignment.<n>Within this framework, we present a taxonomy of reward mechanisms, analyze reward hacking as a pervasive failure mode, and examine how reward signals unify challenges.
  arXiv  Detail & Related papers  (2026-02-10T00:45:24Z)
• Multimodal Reinforcement Learning with Agentic Verifier for AI Agents [131.46008226323423]
  Argos is a principled multimodal reward agent to train reasoning models for agentic tasks.<n>By leveraging our agentic verifier across both SFT data and RL training, our model achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2025-12-03T04:42:47Z)
• Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning [58.33560203572211]
  We exploit off-the-shelf video diffusion models pretrained on large-scale video datasets.<n>For video-level rewards, we first finetune a pretrained video diffusion model on domain-specific datasets.<n>We then employ a learned forward-backward representation that represents the probability of visiting the goal state from a given state-action pair as frame-level reward.
  arXiv  Detail & Related papers  (2025-11-30T16:22:27Z)
• GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning [46.09328632452354]
  We introduce GRACE, a method for using Large Language Models within an evolutionary search to reverse-engineer an interpretable, code-based reward function.<n>The resulting reward function is executable code that can be inspected and verified.<n>We empirically validate GRACE on the BabyAI and AndroidWorld benchmarks, where it efficiently learns highly accurate rewards.
  arXiv  Detail & Related papers  (2025-10-02T16:31:39Z)
• One RL to See Them All: Visual Triple Unified Reinforcement Learning [92.90120580989839]
  We propose V-Triune, a Visual Triple Unified Reinforcement Learning system that enables visual reasoning and perception tasks within a single training pipeline.<n>V-Triune comprises triple complementary components: Sample-Level Datashelf (to unify diverse task inputs), Verifier-Level Reward (to deliver custom rewards via specialized verifiers).<n>We introduce a novel Dynamic IoU reward, which provides adaptive, progressive, and definite feedback for perception tasks handled by V-Triune.
  arXiv  Detail & Related papers  (2025-05-23T17:41:14Z)
• Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems [54.4392552373835]
  Reward models (RMs) are crucial for the training and inference-time scaling up of large language models (LLMs)<n>We propose agentic reward modeling, a reward system that combines reward models with verifiable correctness signals to provide reliable rewards.<n>We conduct comprehensive experiments on existing reward model benchmarks and inference time best-of-n searches on real-world downstream tasks.
  arXiv  Detail & Related papers  (2025-02-26T17:19:12Z)
• Large Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcement Learning [5.203141160419753]
  We propose ERFSL, an efficient reward function searcher using LLMs. We generate reward components for each numerically explicit user requirement. We employ a reward critic to identify the correct code form.
  arXiv  Detail & Related papers  (2024-09-04T04:15:14Z)
• RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback [24.759613248409167]
  Reward engineering has long been a challenge in Reinforcement Learning research. We propose RL-VLM-F, a method that automatically generates reward functions for agents to learn new tasks. We demonstrate that RL-VLM-F successfully produces effective rewards and policies across various domains.
  arXiv  Detail & Related papers  (2024-02-06T04:06:06Z)
• Deep Reinforcement Learning from Hierarchical Preference Design [99.46415116087259]
  This paper shows by exploiting certain structures, one can ease the reward design process. We propose a hierarchical reward modeling framework -- HERON for scenarios: (I) The feedback signals naturally present hierarchy; (II) The reward is sparse, but with less important surrogate feedback to help policy learning.
  arXiv  Detail & Related papers  (2023-09-06T00:44:29Z)
• Video Prediction Models as Rewards for Reinforcement Learning [127.53893027811027]
  VIPER is an algorithm that leverages pretrained video prediction models as action-free reward signals for reinforcement learning. We see our work as starting point for scalable reward specification from unlabeled videos.
  arXiv  Detail & Related papers  (2023-05-23T17:59:33Z)
• Reinforcement Learning Friendly Vision-Language Model for Minecraft [31.863271032186038]
  We propose a novel cross-modal contrastive learning framework architecture, CLIP4MC. We aim to learn a reinforcement learning (RL) friendly vision-language model (VLM) that serves as an intrinsic reward function for open-ended tasks. We demonstrate that the proposed method achieves better performance on RL tasks compared with baselines.
  arXiv  Detail & Related papers  (2023-03-19T05:20:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.