MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning

• URL: http://arxiv.org/abs/2603.25108v1
• Date: Thu, 26 Mar 2026 07:27:11 GMT
• Title: MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning
• Abstract summary: Multi-Stage Reinforcement Learning (MSRL) can achieve scalable RL for multimodal reward models with limited multimodal data.<n>MSRL replaces the conventional RLVR-based training paradigm by first learning a generalizable reward reasoning capability.<n>Extensive experiments demonstrate that MSRL effectively scales the RLVR-based training of generative MRMs.
• Score: 59.03162187361455
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in multimodal reward modeling have been largely driven by a paradigm shift from discriminative to generative approaches. Building on this progress, recent studies have further employed reinforcement learning from verifiable rewards (RLVR) to enhance multimodal reward models (MRMs). Despite their success, RLVR-based training typically relies on labeled multimodal preference data, which are costly and labor-intensive to obtain, making it difficult to scale MRM training. To overcome this limitation, we propose a Multi-Stage Reinforcement Learning (MSRL) approach, which can achieve scalable RL for MRMs with limited multimodal data. MSRL replaces the conventional RLVR-based training paradigm by first learning a generalizable reward reasoning capability from large-scale textual preference data, and then progressively transferring this capability to multimodal tasks through caption-based and fully multimodal reinforcement-learning stages. Furthermore, we introduce a cross-modal knowledge distillation approach to improve preference generalization within MSRL. Extensive experiments demonstrate that MSRL effectively scales the RLVR-based training of generative MRMs and substantially improves their performance across both visual understanding and visual generation tasks (e.g., from 66.6% to 75.9% on VL-RewardBench and from 70.2% to 75.7% on GenAI-Bench), without requiring additional multimodal preference annotations. Our code is available at: https://github.com/wangclnlp/MSRL.

Related papers

• Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation [21.111047537783563]
  We present MIFS, a systematic pipeline designed to generate RL-ready multimodal data.<n>We show that MIFS-trained MLLMs achieve an average improvement of 8.13% on four MMIF benchmarks and a 3$times$ faster training convergence compared to using raw data.
  arXiv  Detail & Related papers  (2026-09-13T08:50:30Z)
• Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL [55.05743310621117]
  Co-RL is a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers.<n>We show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops.
  arXiv  Detail & Related papers  (2026-08-18T01:16:02Z)
• OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe [69.90298686714036]
  We introduce OpenMMReasoner, a fully transparent two-stage recipe for multimodal reasoning spanning fine-tuning and reinforcement learning.<n>Our method achieves a 11.6% improvement over the Qwen2.5-VL-7B-Instruct baseline across nine multimodal reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-11-20T13:11:45Z)
• Reinforcement Learning on Pre-Training Data [55.570379963147424]
  We introduce Reinforcement Learning on Pre-Training data (R), a new training-time scaling paradigm for optimizing large language models (LLMs)<n>R enables the policy to autonomously explore meaningful trajectories to learn from pre-training data and improve its capability through reinforcement learning (RL)<n>Extensive experiments on both general-domain and mathematical reasoning benchmarks across multiple models validate the effectiveness of R.
  arXiv  Detail & Related papers  (2025-09-23T17:10:40Z)
• Your Reward Function for RL is Your Best PRM for Search: Unifying RL and Search-Based TTS [62.22644307952087]
  We introduce AIRL-S, the first natural unification of RL-based and search-based TTS.<n>We leverage adversarial inverse reinforcement learning (AIRL) combined with group relative policy optimization (GRPO) to learn a dense, dynamic PRM directly from correct reasoning traces.<n>Our results show that our unified approach improves performance by 9 % on average over the base model, matching GPT-4o.
  arXiv  Detail & Related papers  (2025-08-19T23:41:15Z)
• MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancement [37.880962254812175]
  Multi-Expert Mutual Learning GRPO is an innovative framework that utilizes diverse expert prompts as system prompts to generate a broader range of responses.<n>We show that MEML- GRPO delivers significant improvements, achieving an average performance gain of 4.89% with Qwen and 11.33% with Llama.
  arXiv  Detail & Related papers  (2025-08-13T09:58:10Z)
• MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization [74.04867639197445]
  MiroMind-M1 is a set of fully open-source RLMs built on the Qwen-2.5-based benchmarks.<n>Our models are trained in two stages: SFT on a carefully curated corpus of 719K math-reasoning problems with verified CoT trajectories, followed by RLVR on 62K challenging and verifiable problems.
  arXiv  Detail & Related papers  (2025-07-19T16:21:23Z)
• Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning [28.92744927199283]
  ReVisual-R1 achieves a new state-of-the-art among open-source 7B MLLMs on challenging benchmarks including MathVerse, MathVision, WeMath, LogicVista, DynaMath, and challenging AIME2024 and AIME2025.
  arXiv  Detail & Related papers  (2025-06-04T17:51:08Z)
• R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning [22.167272219418845]
  Multimodal Reward Models (MRMs) play a crucial role in enhancing the performance of Multimodal Large Language Models (MLLMs)<n>We propose the StableReinforce algorithm, which refines the training loss, advantage estimation strategy, and reward design of existing RL methods.<n>Our reward model, R1-Reward, trained using the StableReinforce algorithm on this dataset, significantly improves performance on multimodal reward modeling benchmarks.
  arXiv  Detail & Related papers  (2025-05-05T17:59:50Z)
• OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learning [29.053899071144976]
  We propose OThink-MR1, an advanced MLLM equipped with profound comprehension and reasoning capabilities across multimodal tasks.<n>Specifically, we introduce Group Relative Policy Optimization with a dynamic Kullback-Leibler strategy.<n> GRPO-D achieves a relative improvement of more than 5.72% over SFT and more than 13.59% over GRPO in same-task evaluation.
  arXiv  Detail & Related papers  (2025-03-20T12:22:18Z)
• MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning [55.82649731348012]
  We introduce the MMK12 dataset and MM-EUREKA with 7B and 32B parameters.<n>The former is a high-quality multimodal mathematics reasoning dataset featuring diverse knowledge domains with human-verified answers and solution processes.<n>The latter is a multimodal model employing rule-based reinforcement learning utilizing online filtering and two-stage training strategy to enhance training stability.
  arXiv  Detail & Related papers  (2025-03-10T14:23:12Z)
• Effective Multimodal Reinforcement Learning with Modality Alignment and Importance Enhancement [41.657470314421204]
  It is challenging to train an agent via reinforcement learning due to the heterogeneity and dynamic importance of different modalities. We propose a novel multimodal RL approach that can do multimodal alignment and importance enhancement according to their similarity and importance. We test our approach on several multimodal RL domains, showing that it outperforms state-of-the-art methods in terms of learning speed and policy quality.
  arXiv  Detail & Related papers  (2023-02-18T12:35:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.