To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models

• URL: http://arxiv.org/abs/2602.12566v1
• Date: Fri, 13 Feb 2026 03:25:13 GMT
• Title: To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
• Abstract summary: Reinforcement Learning with Verifiable Rewards (RLVR) plays a key role in stimulating the explicit reasoning capability of Large Language Models (LLMs)<n>We design extensive qualitative and quantitative experiments using open-source datasets.<n>We find the RLVR across domains exhibits few mutual interferences, and reasoning-intensive domains demonstrate mutually synergistic effects.
• Score: 36.65680320664014
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) plays a key role in stimulating the explicit reasoning capability of Large Language Models (LLMs). We can achieve expert-level performance in some specific domains via RLVR, such as coding or math. When a general multi-domain expert-level model is required, we need to carefully consider the collaboration of RLVR across different domains. The current state-of-the-art models mainly employ two different training paradigms for multi-domain RLVR: mixed multi-task RLVR and separate RLVR followed by model merging. However, most of the works did not provide a detailed comparison and analysis about these paradigms. To this end, we choose multiple commonly used high-level tasks (e.g., math, coding, science, and instruction following) as our target domains and design extensive qualitative and quantitative experiments using open-source datasets. We find the RLVR across domains exhibits few mutual interferences, and reasoning-intensive domains demonstrate mutually synergistic effects. Furthermore, we analyze the internal mechanisms of mutual gains from the perspectives of weight space geometry, model prediction behavior, and information constraints. This project is named as M2RL that means Mixed multi-task training or separate training followed by model Merging for Reinforcement Learning, and the homepage is at https://github.com/mosAI25/M2RL

Related papers

• Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms [50.28963615440865]
  We organize three fusion paradigms by the artefacts they reuse.<n>We compare all three using shared experts and data across model scales and a multi-domain benchmark suite.
  arXiv  Detail & Related papers  (2026-08-27T17:38:04Z)
• Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning [82.91265691530351]
  A$2$D is an Adaptive Ability Decomposing method for enhancing the effectiveness ofReinforcement Learning with verifiable rewards.<n>We first train a decomposer via RLVR without distillation, enabling it to decompose complex questions into a set of simpler sub-questions.<n>Next, we use this decomposer to annotate sub-questions for each question in the training dataset, and then train the reasoner under RLVR with sub-question guidance.
  arXiv  Detail & Related papers  (2026-01-31T14:48:23Z)
• Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development [32.14985932997508]
  Diffusion Models (DMs) offer key advantages for reinforcement learning (RL)<n>This survey delivers a comprehensive and up-to-date synthesis of diffusion-based RL.
  arXiv  Detail & Related papers  (2025-10-14T08:03:46Z)
• WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning [17.459985667824807]
  Building on the success of text-based reasoning models like DeepSeek-R1, extending these capabilities to multimodal reasoning holds great promise.<n>In this paper, we show how to achieve the general-purpose visual-language reasoning through reinforcement learning.
  arXiv  Detail & Related papers  (2025-06-09T16:20:54Z)
• MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning [37.71233459623324]
  Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a powerful paradigm for post-training large language models (LLMs)<n>Applying RLVR to Multimodal LLMs (MLLMs) presents significant opportunities but is complicated by the broader, heterogeneous nature of vision-language tasks.<n>We introduce a systematic post-training framework for Multimodal LLM RLVR, featuring a rigorous data mixture problem formulation and benchmark implementation.
  arXiv  Detail & Related papers  (2025-05-30T17:59:38Z)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging [124.91183814854126]
  Model merging seeks to combine multiple expert models into a single model.<n>We introduce a benchmark for model merging research that clearly divides the tasks for MLLM training and evaluation.<n>We find that model merging offers a promising way for building improved MLLMs without requiring training data.
  arXiv  Detail & Related papers  (2025-05-26T12:23:14Z)
• Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains [92.36624674516553]
  Reinforcement learning with verifiable rewards (RLVR) has demonstrated significant success in enhancing mathematical reasoning and coding performance of large language models (LLMs)<n>We investigate the effectiveness and scalability of RLVR across diverse real-world domains including medicine, chemistry, psychology, economics, and education.<n>We utilize a generative scoring technique that yields soft, model-based reward signals to overcome limitations posed by binary verifications.
  arXiv  Detail & Related papers  (2025-03-31T08:22:49Z)
• From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons [85.99268361356832]
  We introduce a process of adapting an MLLM to a Generalist Embodied Agent (GEA)<n>GEA is a single unified model capable of grounding itself across varied domains through a multi-embodiment action tokenizer.<n>Our findings reveal the importance of training with cross-domain data and online RL for building generalist agents.
  arXiv  Detail & Related papers  (2024-12-11T15:06:25Z)
• An Empirical Study of Multimodal Model Merging [148.48412442848795]
  Model merging is a technique that fuses multiple models trained on different tasks to generate a multi-task solution. We conduct our study for a novel goal where we can merge vision, language, and cross-modal transformers of a modality-specific architecture. We propose two metrics that assess the distance between weights to be merged and can serve as an indicator of the merging outcomes.
  arXiv  Detail & Related papers  (2023-04-28T15:43:21Z)
• HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression [53.90578309960526]
  Large pre-trained language models (PLMs) have shown overwhelming performances compared with traditional neural network methods. We propose a hierarchical relational knowledge distillation (HRKD) method to capture both hierarchical and domain relational information.
  arXiv  Detail & Related papers  (2021-10-16T11:23:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.