MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

• URL: http://arxiv.org/abs/2606.30406v1
• Date: Mon, 29 Jun 2026 14:51:28 GMT
• Title: MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
• Abstract summary: Multi-teacher On-Policy Distillation (MOPD) is a post-training paradigm for combining the capabilities of multiple domain RL teachers.<n>On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines.
• Score: 29.71786876690702
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.

Related papers

• Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs [13.74158437063828]
  Multi-Teacher Self-Distillation Policy Optimization (MT-SDPO) is an on-policy distillation method that unifies several frozen teachers into one student model.<n>MT-SDPO consists of three components: self-anchors, answer-verified eligibility, and privileged distillation.<n>It lifts the weakest domain of Qwen3-8B by 14.79 points and narrows its domain gap by 74.7%, a better balance than serving one matched teacher per domain.
  arXiv  Detail & Related papers  (2026-09-02T13:00:45Z)
• Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms [50.28963615440865]
  We organize three fusion paradigms by the artefacts they reuse.<n>We compare all three using shared experts and data across model scales and a multi-domain benchmark suite.
  arXiv  Detail & Related papers  (2026-08-27T17:38:04Z)
• D$^3$-MOPD: Adaptive Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation [50.89072318748398]
  Multi-teacher on-policy distillation distills several domain-expert teachers into a single student.<n>Existing approaches typically fix the per-domain data mixture before training.<n>A fixed mixture wastes compute on fast-converging domains and undertrains slower-converging ones.<n>We propose D$3$-MOPD, a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online.
  arXiv  Detail & Related papers  (2026-08-25T17:57:11Z)
• Weak-to-Strong On-Policy Distillation [21.651421588571296]
  We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a framework that improves the strong student by distilling from multiple weak models.<n>Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker.
  arXiv  Detail & Related papers  (2026-07-28T20:31:48Z)
• D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models [27.90064267850009]
  We propose D-OPSD, a novel training paradigm for step-distilled diffusion models.<n>D-OPSD enables the model to learn new concept, style, etc. without sacrificing the original few-step capacity.
  arXiv  Detail & Related papers  (2026-05-06T17:59:34Z)
• MOA: Multi-Objective Alignment for Role-Playing Agents [68.67837397211903]
  We present MOA, a reinforcement-learning framework that enables multi-dimensional, fine-grained rubric optimization for general RPAs.<n>To address the issues of model output diversity and quality, we have also employed thought-augmented rollout with off-policy guidance.<n>This demonstrates the great potential of MOA in building RPAs that can simultaneously meet the demands of role knowledge, persona style, diverse scenarios, and complex multi-turn conversations.
  arXiv  Detail & Related papers  (2025-12-10T15:35:11Z)
• AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework [76.96794548655292]
  Large language models (LLMs) have sparked growing interest in building generalist agents that can learn through online interactions.<n>Applying reinforcement learning (RL) to train LLM agents in multi-turn, multi-task settings remains challenging due to lack of scalable infrastructure and stable training algorithms.<n>We present the AgentRL framework for scalable multi-turn, multi-task agentic RL training.
  arXiv  Detail & Related papers  (2025-10-05T13:40:01Z)
• PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning [54.73049408950049]
  We propose a Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning.<n>Our approach improves unified multimodal retrieval from both structural and learning perspectives.
  arXiv  Detail & Related papers  (2025-07-10T16:47:25Z)
• MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning [26.736078756799635]
  We introduce a new post-training paradigm MAPoRL (Multi-Agent Post-co-training for collaborative LLMs with Reinforcement Learning)<n>In MAPoRL, multiple LLMs first generate their own responses independently and engage in a multi-turn discussion to collaboratively improve the final answer.<n>A MAPoRL verifier evaluates both the answer and the discussion, by assigning a score that verifies the correctness of the answer.<n>The score serves as the co-training reward, and is then maximized through multi-agent RL.
  arXiv  Detail & Related papers  (2025-02-25T18:33:48Z)
• MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning [17.960185808572582]
  MoDULA is a paradigm for improved fine-tuning and parameter efficiency in multi-task learning.<n>MoDULA-Res is a new method within the MoDULA paradigm, which maintains the model's general capability by connecting universal and task-specific experts.
  arXiv  Detail & Related papers  (2024-12-10T10:55:57Z)
• MALT: Improving Reasoning with Multi-Agent LLM Training [67.76186488361685]
  MALT (Multi-Agent LLM Training) is a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps.<n>On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively.
  arXiv  Detail & Related papers  (2024-12-02T19:30:36Z)
• MADiff: Offline Multi-agent Learning with Diffusion Models [79.18130544233794]
  MADiff is a diffusion-based multi-agent learning framework.<n>It works as both a decentralized policy and a centralized controller.<n>Our experiments demonstrate that MADiff outperforms baseline algorithms across various multi-agent learning tasks.
  arXiv  Detail & Related papers  (2023-05-27T02:14:09Z)
• MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning [61.28547338576706]
  Population-based multi-agent reinforcement learning (PB-MARL) refers to the series of methods nested with reinforcement learning (RL) algorithms. We present MALib, a scalable and efficient computing framework for PB-MARL.
  arXiv  Detail & Related papers  (2021-06-05T03:27:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.