Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models

• URL: http://arxiv.org/abs/2605.07244v1
• Date: Fri, 08 May 2026 05:01:40 GMT
• Title: Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models
• Abstract summary: We introduce Mutual Reinforcement Learning, a framework for concurrent RL post-training.<n>The framework combines a Shared Experience Exchange (SEE), Multi-Worker Resource Allocation (MWRA), and a Tokenizer Heterogeneity Layer (THL)<n>We instantiate three controlled probes on top of GRPO: data-level rollout sharing via Peer Rollout Pooling (PRP), value-level advantage sharing via Cross-Policy GRPO Advantage Sharing (XGRPO), and outcome-level success transfer via Success-Gated Transfer (SGT)
• Score: 48.79781352398832
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Mutual Reinforcement Learning, a framework for concurrent RL post-training in which heterogeneous LLM policies exchange typed experience while keeping separate parameters, objectives, and tokenizers. The framework combines a Shared Experience Exchange (SEE), Multi-Worker Resource Allocation (MWRA), and a Tokenizer Heterogeneity Layer (THL) that retokenizes text and aligns token-level traces across incompatible vocabularies. This substrate makes the experience-sharing design question operational across model families. We instantiate three controlled probes on top of GRPO: data-level rollout sharing via Peer Rollout Pooling (PRP), value-level advantage sharing via Cross-Policy GRPO Advantage Sharing (XGRPO), and outcome-level success transfer via Success-Gated Transfer (SGT). A contextual-bandit analysis characterizes their structural positions on a stability-support trade-off: PRP pays density-ratio variance and THL residual costs, XGRPO preserves on-policy actor support while changing scalar baselines, and SGT supplies a rescue-set score direction toward verified peer successes. In the evaluated regime, outcome-level sharing occupies the favorable point of this trade-off.

Related papers

• CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization [21.755134237801443]
  CoRT is a token-level credit weighting method for rubric-conditioned GRPO.<n>We show that CoRT improves over matched response-level GRPO in the vast majority of comparisons.
  arXiv  Detail & Related papers  (2026-07-28T12:45:19Z)
• LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks [89.13372484640392]
  We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach.<n>EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks.<n>These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.
  arXiv  Detail & Related papers  (2026-07-20T16:08:49Z)
• TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning [10.917273110894469]
  Agentic reinforcement learning requires assigning credit to environment-facing actions.<n>Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens.<n>We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit.
  arXiv  Detail & Related papers  (2026-06-30T17:48:07Z)
• Injecting Distributional Awareness into MLLMs via Reinforcement Learning for Deep Imbalanced Regression [15.47617293460676]
  Multimodal large language models (MLLMs) struggle with numerical regression under long-tailed target distributions.<n>We propose a distribution-aware reinforcement learning framework based on Group Relative Policy Optimization.<n>Experiments on a unified suite of long-tailed regression benchmarks show consistent improvements over SFT and existing MLLM regression methods.
  arXiv  Detail & Related papers  (2026-05-02T11:49:03Z)
• When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO [18.988527161000203]
  Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models.<n>We present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples.<n>We propose Bilateral Context Conditioning (BICC), a mechanism that allows the model to cross-reference successful and failed reasoning traces.
  arXiv  Detail & Related papers  (2026-03-13T16:25:02Z)
• Heterogeneous Agent Collaborative Reinforcement Learning [52.99813668995983]
  Heterogeneous Agent Collaborative Reinforcement Learning (HACRL)<n>Building on this paradigm, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer.<n>Experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO by an average of 3.3% while using only half the rollout cost.
  arXiv  Detail & Related papers  (2026-03-03T05:09:49Z)
• CoT-Saliency: Unified Chain-of-Thought Reasoning for Heterogeneous Saliency Tasks [96.64597365827046]
  We present the first unified framework that jointly handles three operationally heterogeneous saliency tasks.<n>We introduce a Chain-of-Thought (CoT) reasoning process in a Vision-Language Model (VLM) to bridge task heterogeneity.<n>We show our model matches or outperforms specialized SOTA methods and strong closed-source VLMs across all tasks.
  arXiv  Detail & Related papers  (2025-11-01T04:37:01Z)
• Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents [90.45197506653341]
  Large language model (LLM) agents increasingly rely on external tools such as search engines to solve complex, multi-step problems.<n>The trajectories of search agents are structurally heterogeneous, where variations in the number, placement, and outcomes of search calls lead to fundamentally different answer directions and reward distributions.<n>Standard policy gradient methods, which use a single global baseline, suffer from what we identify and formalize as cross-stratum bias.<n>We propose Stratified GRPO, whose central component, Stratified Advantage Normalization (SAN), partitions trajectories into homogeneous strata based on their structural properties and computes advantages locally within each
  arXiv  Detail & Related papers  (2025-10-07T17:59:13Z)
• Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition [52.232968183793986]
  General Policy Composition (GPC) is a training-free method that enhances performance by combining the distributional scores of multiple pre-trained policies.<n>GPC consistently improves performance and adaptability across a diverse set of tasks.
  arXiv  Detail & Related papers  (2025-10-01T16:05:53Z)
• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources [113.33902847941941]
  Variance-Aware Sampling (VAS) is a data selection strategy guided by Variance Promotion Score (VPS)<n>We release large-scale, carefully curated resources containing 1.6M long CoT cold-start data and 15k RL QA pairs.<n> Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS.
  arXiv  Detail & Related papers  (2025-09-25T14:58:29Z)
• LLMs Can't Handle Peer Pressure: Crumbling under Multi-Agent Social Interactions [35.71511502901056]
  Large language models (LLMs) are increasingly deployed in multi-agent systems as components of collaborative intelligence.<n>We examine how LLMs form trust from previous impressions, resist misinformation, and integrate peer input during interaction.<n>We present KAIROS, a benchmark simulating quiz contests with peer agents of varying reliability.
  arXiv  Detail & Related papers  (2025-08-24T09:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.