Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models

• URL: http://arxiv.org/abs/2605.25443v1
• Date: Mon, 25 May 2026 05:42:57 GMT
• Title: Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models
• Abstract summary: Post-training has significantly enhanced the reasoning capability of Large Reasoning Models (LRMs)<n> GRPO-style RL methods in multi-domain settings often fail to achieve consistent improvements across all domains due to inherent interference in policy optimization.<n>We propose Multi-domain Contrastive Policy Optimization (MCPO), which analyzes the structural relationships among rollouts and promotes cross-domain knowledge sharing.
• Score: 38.65301275836433
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Post-training has significantly enhanced the reasoning capability of Large Reasoning Models (LRMs), especially with Reinforcement Learning (RL) like Group Relative Policy Optimization (GRPO). However, GRPO-style RL methods in multi-domain settings often fail to achieve consistent improvements across all domains due to inherent interference in policy optimization. Prior studies on multi-domain RL primarily focus on alleviating cross-domain interference, while often neglecting the pivotal role of knowledge sharing, which we argue is the key to transforming cross-domain interactions from harmful competition into beneficial transfer. To address this limitation, we propose Multi-domain Contrastive Policy Optimization (MCPO), which analyzes the structural relationships among rollouts and promotes cross-domain knowledge sharing and in-domain knowledge consolidation in a contrastive manner. Specifically, for a given prompt, MCPO identifies transferable reasoning trajectories from other domains as positive examples, while treating incorrect rollouts as negative ones. It then encourages consistent representations for positive pairs and pushes negative pairs apart, thereby facilitating knowledge transfer and reducing interference. Moreover, MCPO aligns intra-domain correct rollouts to build a consolidated representation space. In this way, MCPO contrastively learns a harmonious representation space that can accommodate diverse multi-domain knowledge. Empirical results show that MCPO improves the reasoning capabilities of LRMs across multiple domains and even outperforms single-domain training in some cases. Code is available at https://github.com/Maricalce/MCPO.

Related papers

• FeDecider: An LLM-Based Framework for Federated Cross-Domain Recommendation [75.50721642765994]
  Large language model (LLM)-based recommendation models have demonstrated impressive performance.<n>We propose an LLM-based framework for Federated cross-domain recommendation, FeDecider.<n>Extensive experiments across diverse datasets validate the effectiveness of our proposed FeDecider.
  arXiv  Detail & Related papers  (2026-02-17T21:42:28Z)
• LLM-EDT: Large Language Model Enhanced Cross-domain Sequential Recommendation with Dual-phase Training [53.539682966282534]
  Cross-domain Sequential Recommendation (CDSR) has been proposed to enrich user-item interactions by incorporating information from various domains.<n>Despite current progress, the imbalance issue and transition issue hinder further development of CDSR.<n>We propose an LLMs Enhanced Cross-domain Sequential Recommendation with Dual-phase Training (LLM-EDT)
  arXiv  Detail & Related papers  (2025-11-25T05:18:04Z)
• WeaveRec: An LLM-Based Cross-Domain Sequential Recommendation Framework with Model Merging [24.949880939628386]
  We introduce WeaveRec, which cross-trains multiple LoRA modules with source and target domain data in a weaving fashion.<n>We provide a theoretical guarantee that WeaveRec can reduce the upper bound of the expected error in the target domain.
  arXiv  Detail & Related papers  (2025-10-30T14:37:15Z)
• MARCO: A Cooperative Knowledge Transfer Framework for Personalized Cross-domain Recommendations [23.945049006150555]
  MARCO is a Multi-Agent Reinforcement Learning-based Cross-Domain recommendation framework.<n>It leverages cooperative multi-agent reinforcement learning, where each agent is dedicated to estimating the contribution from an individual source domain.<n>Experiments across four benchmark datasets demonstrate MARCO's superior performance over state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-10-06T05:49:47Z)
• CDC: Causal Domain Clustering for Multi-Domain Recommendation [32.945861240561]
  Multi-domain recommendation leverages domain-general knowledge to improve recommendations across several domains.<n>Existing domain grouping methods, based on business logic or data similarities, often fail to capture the true transfer relationships required for optimal grouping.<n>We propose Causal Domain Clustering (CDC) to effectively cluster domains.<n>CDC significantly enhances performance across over 50 domains on public datasets and in industrial settings, achieving a 4.9% increase in online eCPM.
  arXiv  Detail & Related papers  (2025-07-09T14:15:47Z)
• DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data [65.09939942413651]
  We propose a principled extension to GRPO that addresses inter-group imbalance with two key innovations.<n> Domain-aware reward scaling counteracts frequency bias by reweighting optimization based on domain prevalence.<n>Difficulty-aware reward scaling leverages prompt-level self-consistency to identify and prioritize uncertain prompts that offer greater learning value.
  arXiv  Detail & Related papers  (2025-05-21T03:43:29Z)
• Cross-Domain Policy Transfer by Representation Alignment via Multi-Domain Behavioral Cloning [13.674493608667627]
  We present a simple approach for cross-domain policy transfer that learns a shared latent representation across domains and a common abstract policy on top of it. Our approach leverages multi-domain behavioral cloning on unaligned trajectories of proxy tasks and employs maximum mean discrepancy (MMD) as a regularization term to encourage cross-domain alignment.
  arXiv  Detail & Related papers  (2024-07-24T00:13:00Z)
• Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Models [55.51408151807268]
  When tailored to specific domains, Large Language Models (LLMs) tend to experience catastrophic forgetting. crafting a versatile model for multiple domains simultaneously often results in a decline in overall performance. We present the RolE Prompting Guided Multi-Domain Adaptation (REGA) strategy.
  arXiv  Detail & Related papers  (2024-03-05T08:22:41Z)
• Learning to Combine: Knowledge Aggregation for Multi-Source Domain Adaptation [56.694330303488435]
  We propose a Learning to Combine for Multi-Source Domain Adaptation (LtC-MSDA) framework. In the nutshell, a knowledge graph is constructed on the prototypes of various domains to realize the information propagation among semantically adjacent representations. Our approach outperforms existing methods with a remarkable margin.
  arXiv  Detail & Related papers  (2020-07-17T07:52:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.