Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation

• URL: http://arxiv.org/abs/2609.16059v1
• Date: Sun, 13 Sep 2026 08:50:30 GMT
• Title: Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation
• Abstract summary: We present MIFS, a systematic pipeline designed to generate RL-ready multimodal data.<n>We show that MIFS-trained MLLMs achieve an average improvement of 8.13% on four MMIF benchmarks and a 3$times$ faster training convergence compared to using raw data.
• Score: 21.111047537783563
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multimodal instruction following (MMIF) is crucial for building generalist agents. However, current training paradigms rely heavily on Supervised Fine-Tuning (SFT), which often leads to surface-level pattern matching and degrades general capabilities. While Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising alternative, its scalability in MMIF is severely bottlenecked by the scarcity of high-quality, RL-ready multimodal data. To bridge this gap, we present MIFS (\textbf{M}ultimodal \textbf{I}nstruction \textbf{F}ollowing \textbf{S}ynthesis), a systematic pipeline designed to generate RL-ready multimodal data. Specifically, MIFS introduces a generative constraint protocol to synthesize diverse raw samples, followed by a learnability-aware distillation mechanism that filters data based on RL training dynamics to ensure stable policy optimization. Furthermore, a code-based verifier provides high-precision reward signals for policy learning. The resulting dataset comprises 90k samples across 8 constraint categories and 14 task domains. Empirical evaluations demonstrate that MIFS-trained MLLMs achieve an average improvement of 8.13\% on four MMIF benchmarks and a 3$\times$ faster training convergence compared to using raw data. Crucially, our approach mitigates the generalization trade-offs typical of SFT, preserving core visual capabilities while significantly boosting instruction-following precision.

Related papers

• Unified Data Selection for LLM Reasoning [66.0118347918729]
  Training Large Language Models (LLMs) for complex, long-CoT reasoning is often bottlenecked by the need for massive high-quality reasoning data.<n>We propose High-Entropy Sum (HES), a training-free metric that quantifies reasoning quality by summing only the entropy of the top (e.g., 0.5%) highest-entropy tokens in each reasoning sample.
  arXiv  Detail & Related papers  (2026-05-21T12:21:41Z)
• MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning [59.03162187361455]
  Multi-Stage Reinforcement Learning (MSRL) can achieve scalable RL for multimodal reward models with limited multimodal data.<n>MSRL replaces the conventional RLVR-based training paradigm by first learning a generalizable reward reasoning capability.<n>Extensive experiments demonstrate that MSRL effectively scales the RLVR-based training of generative MRMs.
  arXiv  Detail & Related papers  (2026-03-26T07:27:11Z)
• Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration [56.074760766965085]
  PRISM achieves a dynamics-aware framework that arbitrates data based on its degree of cognitive conflict with the model's existing knowledge.<n>Our findings suggest that disentangling data based on internal optimization regimes is crucial for scalable and robust agent alignment.
  arXiv  Detail & Related papers  (2026-01-12T05:43:20Z)
• RS-Prune: Training-Free Data Pruning at High Ratios for Efficient Remote Sensing Diffusion Foundation Models [14.093802378976315]
  Diffusion-based remote sensing (RS) generative foundation models rely on large amounts of globally representative data.<n>We propose a training-free, two-stage data pruning approach that quickly select a high-quality subset under high pruning ratios.<n> Experiments show that, even after pruning 85% of the training data, our method significantly improves convergence and generation quality.
  arXiv  Detail & Related papers  (2025-12-29T06:44:06Z)
• OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe [69.90298686714036]
  We introduce OpenMMReasoner, a fully transparent two-stage recipe for multimodal reasoning spanning fine-tuning and reinforcement learning.<n>Our method achieves a 11.6% improvement over the Qwen2.5-VL-7B-Instruct baseline across nine multimodal reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-11-20T13:11:45Z)
• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources [113.33902847941941]
  Variance-Aware Sampling (VAS) is a data selection strategy guided by Variance Promotion Score (VPS)<n>We release large-scale, carefully curated resources containing 1.6M long CoT cold-start data and 15k RL QA pairs.<n> Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS.
  arXiv  Detail & Related papers  (2025-09-25T14:58:29Z)
• Reinforcement Learning on Pre-Training Data [55.570379963147424]
  We introduce Reinforcement Learning on Pre-Training data (R), a new training-time scaling paradigm for optimizing large language models (LLMs)<n>R enables the policy to autonomously explore meaningful trajectories to learn from pre-training data and improve its capability through reinforcement learning (RL)<n>Extensive experiments on both general-domain and mathematical reasoning benchmarks across multiple models validate the effectiveness of R.
  arXiv  Detail & Related papers  (2025-09-23T17:10:40Z)
• A Distribution-Aware Flow-Matching for Generating Unstructured Data for Few-Shot Reinforcement Learning [1.0709300917082865]
  We introduce a distribution-aware flow matching approach to generate synthetic unstructured data for few-shot reinforcement learning.<n>Our approach addresses key challenges in traditional model-based RL, such as overfitting and data correlation.<n>Results demonstrate that our method achieves stable convergence in terms of maximum Q-value while enhancing frame rates by 30% in the initial timestamps.
  arXiv  Detail & Related papers  (2024-09-21T15:50:59Z)
• MCDFN: Supply Chain Demand Forecasting via an Explainable Multi-Channel Data Fusion Network Model [0.0]
  We introduce the Multi-Channel Data Fusion Network (MCDFN), a hybrid architecture that integrates CNN, Long Short-Term Memory networks (LSTM), and Gated Recurrent Units (GRU)<n>Our comparative benchmarking demonstrates that MCDFN outperforms seven other deep-learning models.<n>This research advances demand forecasting methodologies and offers practical guidelines for integrating MCDFN into supply chain systems.
  arXiv  Detail & Related papers  (2024-05-24T14:30:00Z)
• Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization [165.98557106089777]
  A key challenge is to enhance the capabilities of large language models (LLMs) amid a looming shortage of high-quality training data. Our study starts from an empirical strategy for the light continual training of LLMs using their original pre-training data sets. We then formalize this strategy into a principled framework of Instance-Reweighted Distributionally Robust Optimization.
  arXiv  Detail & Related papers  (2024-02-22T04:10:57Z)
• Task Aware Modulation using Representation Learning: An Approach for Few Shot Learning in Environmental Systems [15.40286222692196]
  TAM-RL is a novel framework for few-shot learning in heterogeneous systems. We evaluate TAM-RL on two real-world environmental datasets.
  arXiv  Detail & Related papers  (2023-10-07T07:55:22Z)
• Multifidelity Reinforcement Learning with Control Variates [3.2895195535353317]
  In many computational science and engineering applications, the output of a system of interest corresponding to a given input can be queried at different levels of fidelity with different costs. We study the reinforcement learning problem in the presence of multiple environments with different levels of fidelity for a given control task. A multifidelity estimator that exploits the cross-correlations between the low- and high-fidelity returns is proposed to reduce the variance in the estimation of the state-action value function.
  arXiv  Detail & Related papers  (2022-06-10T15:01:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.