$n$-Musketeers: Reinforcement Learning Shapes Collaboration Among Language Models

• URL: http://arxiv.org/abs/2602.09173v1
• Date: Mon, 09 Feb 2026 20:27:52 GMT
• Title: $n$-Musketeers: Reinforcement Learning Shapes Collaboration Among Language Models
• Abstract summary: We introduce soft hidden-state collaboration, where multiple frozen SLM experts are integrated via a trainable attention interface.<n>Experiments on Reasoning Gym and GSM8K show that this latent integration is competitive with strong single-model RLVR baselines.<n>Overall, hidden-state collaboration provides a compact mechanism for leveraging frozen experts, while offering an observational window into expert utilization patterns and their evolution under RLVR.
• Score: 17.293403543585413
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent progress in reinforcement learning with verifiable rewards (RLVR) shows that small, specialized language models (SLMs) can exhibit structured reasoning without relying on large monolithic LLMs. We introduce soft hidden-state collaboration, where multiple heterogeneous frozen SLM experts are integrated through their internal representations via a trainable attention interface. Experiments on Reasoning Gym and GSM8K show that this latent integration is competitive with strong single-model RLVR baselines. Ablations further reveal a dual mechanism of expert utilization: for simpler arithmetic domains, performance gains can largely be explained by static expert preferences, whereas more challenging settings induce increasingly concentrated and structured expert attention over training, indicating emergent specialization in how the router connects to relevant experts. Overall, hidden-state collaboration provides a compact mechanism for leveraging frozen experts, while offering an observational window into expert utilization patterns and their evolution under RLVR.

Related papers

• Rethinking Expert Training for Model Merging with Prompt Learning [29.003417422329715]
  Model merging aims to combine multiple domain-specialized experts trained from a shared foundation model into a single multi-task model.<n>Existing approaches largely focus on improving the merging procedure itself.<n>We introduce Dual-Tuned Experts (DTEs), a two-stage training strategy that first learns prompts and then fine-tunes the vision encoder.
  arXiv  Detail & Related papers  (2026-07-27T14:04:48Z)
• Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild [49.77540427384148]
  We present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild.<n>To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms.<n>AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings.
  arXiv  Detail & Related papers  (2026-07-15T14:30:20Z)
• Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads [48.341678148905466]
  We present an in-depth interpretability study that uncovers a profound structural property within Multimodal Large Language Models (MLLMs)<n>We identify and characterize a highly specialized subset of attention heads, referred to as Context-aware Retrieval (CoRe) heads.<n>Our findings reveal a structural principle of functional sparsity within MLLMs, refining the current understanding of mechanistic interpretability.
  arXiv  Detail & Related papers  (2026-06-04T08:18:31Z)
• PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization [25.22639372242625]
  We introduce textbfPRISM, a novel dual-stream Mixture-of-Experts (MoE) framework that synergizes VFMs via modular specialization.<n>Experiments on PASCAL-Context and NYUD-v2 show that textbfPRISM establishes a new state of the art, validating that sparse, emergent specialization is a scalable approach for integrating diverse visual knowledge.
  arXiv  Detail & Related papers  (2026-06-02T10:28:32Z)
• ERNIE 5.0 Technical Report [244.36480708815316]
  ERNIE 5.0 is a unified autoregressive foundation model for unified multimodal understanding and generation across text, image, video, and audio.<n>To address practical challenges in large-scale deployment under diverse resource constraints, ERNIE 5.0 adopts a novel elastic training paradigm.<n>We show that ERNIE 5.0 achieves strong and balanced performance across multiple modalities.
  arXiv  Detail & Related papers  (2026-02-04T16:18:15Z)
• Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder [59.89996751196727]
  Sparse autoencoders (SAEs) have emerged as a powerful tool for interpreting large language models.<n>SAEs' hidden layers have high dimensionality to satisfy sparsity constraints, resulting in prohibitive training and inference costs.<n>Recent Mixture of Experts (MoE) approaches attempt to address this by SAEs into narrower expert networks with gated activation.<n>We propose two key innovations: (1) Multiple Expert Activation that simultaneously engages semantically weighted expert subsets to encourage specialization, and (2) Feature Scaling that enhances diversity through adaptive high-frequency scaling.
  arXiv  Detail & Related papers  (2025-11-07T22:19:34Z)
• Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms [55.1784306456972]
  Mixture-of-Experts (MoE) architectures have emerged as a promising direction, offering efficiency and scalability by activating only a subset of parameters during inference.<n>We use an internal metric to investigate the mechanisms of MoE architecture by explicitly incorporating routing mechanisms and analyzing expert-level behaviors.<n>We uncover several findings: (1) neuron utilization decreases as models evolve, reflecting stronger generalization; (2) training exhibits a dynamic trajectory, where benchmark performance alone provides limited signal; (3) task completion emerges from collaborative contributions of multiple experts, with shared experts driving concentration; and (4) activation patterns at the neuron level provide a fine-grained proxy for data diversity.
  arXiv  Detail & Related papers  (2025-09-28T15:13:38Z)
• Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time [35.31371938688921]
  Test-Time Scaling (TTS) enhances the reasoning ability of large language models (LLMs) by allocating additional computation during inference.<n>We propose Dynamic Experts Search (DES), a TTS strategy that elevates expert activation into a controllable dimension of the search space.
  arXiv  Detail & Related papers  (2025-09-26T16:49:10Z)
• Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models [5.211806751260724]
  We propose a hierarchical sparse dictionary learning (HSDL) method that uncovers the collaboration patterns among experts.<n>We also introduce the Contribution-Aware Expert Pruning (CAEP) algorithm, which effectively prunes low-contribution experts.
  arXiv  Detail & Related papers  (2025-04-16T04:06:15Z)
• Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains [92.36624674516553]
  Reinforcement learning with verifiable rewards (RLVR) has demonstrated significant success in enhancing mathematical reasoning and coding performance of large language models (LLMs)<n>We investigate the effectiveness and scalability of RLVR across diverse real-world domains including medicine, chemistry, psychology, economics, and education.<n>We utilize a generative scoring technique that yields soft, model-based reward signals to overcome limitations posed by binary verifications.
  arXiv  Detail & Related papers  (2025-03-31T08:22:49Z)
• Astrea: A MOE-based Visual Understanding Model with Progressive Alignment [10.943104653307294]
  Vision-Language Models (VLMs) based on Mixture-of-Experts (MoE) architectures have emerged as a pivotal paradigm in multimodal understanding.<n>We propose Astrea, a novel multi-expert collaborative VLM architecture based on progressive pre-alignment.
  arXiv  Detail & Related papers  (2025-03-12T14:44:52Z)
• Multi-Head Mixture-of-Experts [100.60556163597946]
  We propose Multi-Head Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each token into multiple sub-tokens. MH-MoE is straightforward to implement and decouples from other SMoE optimization methods, making it easy to integrate with other SMoE models for enhanced performance.
  arXiv  Detail & Related papers  (2024-04-23T13:47:09Z)
• Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts [74.40198929049959]
  Large multi-modal models (LMMs) exhibit remarkable performance across numerous tasks. generalist LMMs often suffer from performance degradation when tuned over a large collection of tasks. We propose Omni-SMoLA, an architecture that uses the Soft MoE approach to mix many multimodal low rank experts.
  arXiv  Detail & Related papers  (2023-12-01T23:04:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.