Co-Evolving Policy Distillation

• URL: http://arxiv.org/abs/2604.27083v1
• Date: Wed, 29 Apr 2026 18:24:11 GMT
• Title: Co-Evolving Policy Distillation
• Abstract summary: Co-Evolving Policy Distillation (CoPD) encourages parallel training of experts and introduces OPD during each expert's ongoing RLVR training.<n>Experiments validate that CoPD achieves all-in-one integration of text, image, and video reasoning capabilities.
• Score: 56.500154041818746
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: RLVR and OPD have become standard paradigms for post-training. We provide a unified analysis of these two paradigms in consolidating multiple expert capabilities into a single model, identifying capability loss in different ways: mixed RLVR suffers from inter-capability divergence cost, while the pipeline of first training experts and then performing OPD, though avoiding divergence, fails to fully absorb teacher capabilities due to large behavioral pattern gaps between teacher and student. We propose Co-Evolving Policy Distillation (CoPD), which encourages parallel training of experts and introduces OPD during each expert's ongoing RLVR training rather than after complete expert training, with experts serving as mutual teachers (making OPD bidirectional) to co-evolve. This enables more consistent behavioral patterns among experts while maintaining sufficient complementary knowledge throughout. Experiments validate that CoPD achieves all-in-one integration of text, image, and video reasoning capabilities, significantly outperforming strong baselines such as mixed RLVR and MOPD, and even surpassing domain-specific experts. The model parallel training pattern offered by CoPD may inspire a novel training scaling paradigm.

Related papers

• Graph Neural Team Recommendation: An Integrated Approach [5.347601956905657]
  We reformulate the team recommendation problem into end-to-end link predictions in the expert collaboration graph.<n>Our experiments on two large-scale datasets demonstrate the superiority of the end-to-end approach.
  arXiv  Detail & Related papers  (2026-08-10T22:23:32Z)
• Rethinking Expert Training for Model Merging with Prompt Learning [29.003417422329715]
  Model merging aims to combine multiple domain-specialized experts trained from a shared foundation model into a single multi-task model.<n>Existing approaches largely focus on improving the merging procedure itself.<n>We introduce Dual-Tuned Experts (DTEs), a two-stage training strategy that first learns prompts and then fine-tunes the vision encoder.
  arXiv  Detail & Related papers  (2026-07-27T14:04:48Z)
• EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning [77.47708145025832]
  EvoTrainer is an autonomous training framework that co-evolves policies and training-side harnesses.<n>It diagnoses rollout-level evidence, revises diagnostics, backtests interventions, and accumulates reusable skills.
  arXiv  Detail & Related papers  (2026-06-02T03:47:48Z)
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe [53.40076304466524]
  We propose Uni-OPD, a unified OPD framework that generalizes across Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs)<n>Specifically, from the student's perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training.<n>We develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories.
  arXiv  Detail & Related papers  (2026-05-05T12:15:21Z)
• Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner [91.12249411043723]
  In-context reinforcement learning can be used to train agents that can acquire new tasks directly at inference.<n>Decision Pre-Trained Transformer (DPT) was introduced as an alternative, showing stronger in-context reinforcement learning abilities.<n>We extend DPT to diverse multi-domain environments, applying Flow Matching as a natural training choice.
  arXiv  Detail & Related papers  (2026-04-06T19:18:12Z)
• KARL: Knowledge Agents via Reinforcement Learning [63.627906947205624]
  We present a system for training enterprise search agents via reinforcement learning.<n> KARLBench is a multi-capability evaluation suite spanning six distinct search regimes.<n>We show that models trained across heterogeneous search behaviors generalize substantially better than those optimized for any single benchmark.
  arXiv  Detail & Related papers  (2026-03-05T14:30:25Z)
• Heterogeneous Agent Collaborative Reinforcement Learning [52.99813668995983]
  Heterogeneous Agent Collaborative Reinforcement Learning (HACRL)<n>Building on this paradigm, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer.<n>Experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO by an average of 3.3% while using only half the rollout cost.
  arXiv  Detail & Related papers  (2026-03-03T05:09:49Z)
• Continual Policy Distillation from Distributed Reinforcement Learning Teachers [14.879372764916154]
  Continual Reinforcement Learning aims to develop lifelong learning agents to continuously acquire knowledge across diverse tasks.<n>This requires efficiently managing the stability-plasticity dilemma and leveraging prior experience to rapidly generalize to novel tasks.<n>We propose a novel teacher-student framework that decouples CRL into two independent processes: training single-task teacher models through distributed RL and continually distilling them into a central generalist model.
  arXiv  Detail & Related papers  (2026-01-30T02:40:34Z)
• More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration [103.1589018460702]
  "guidance-on-demand" approach expands exploration while preserving the value of self-discovery.<n>Experiments show AMPO substantially outperforms a strong baseline.<n>Using four peer-sized teachers, our method achieves comparable results to approaches that leverage a single, more powerful teacher.
  arXiv  Detail & Related papers  (2025-10-02T17:14:00Z)
• Astrea: A MOE-based Visual Understanding Model with Progressive Alignment [10.943104653307294]
  Vision-Language Models (VLMs) based on Mixture-of-Experts (MoE) architectures have emerged as a pivotal paradigm in multimodal understanding.<n>We propose Astrea, a novel multi-expert collaborative VLM architecture based on progressive pre-alignment.
  arXiv  Detail & Related papers  (2025-03-12T14:44:52Z)
• A Two-Stage Learning-to-Defer Approach for Multi-Task Learning [3.4289478404209826]
  We introduce a novel Two-Stage L2D framework for multi-task learning that integrates classification and regression through a unified deferral mechanism.<n>Our method leverages a two-stage surrogate loss family, which we prove to be both Bayes-consistent and $(mathcalG, mathcalR)$-consistent.
  arXiv  Detail & Related papers  (2024-10-21T07:44:57Z)
• Continuous Control With Ensemble Deep Deterministic Policy Gradients [3.3465375629622502]
  We show how Ensemble Deep Deterministic Policy Gradients (ED2) can yield state-of-the-art results on continuous control tasks from OpenAI Gym MuJoCo. From the practical side, ED2 is conceptually straightforward, easy to code, and does not require knowledge outside of the existing deep reinforcement learning toolbox.
  arXiv  Detail & Related papers  (2021-11-30T13:28:13Z)
• Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control [65.00425082663146]
  We present a Knowledge Transfer based Multi-task Deep Reinforcement Learning framework (KTM-DRL) for continuous control. In KTM-DRL, the multi-task agent first leverages an offline knowledge transfer algorithm to quickly learn a control policy from the experience of task-specific teachers. The experimental results well justify the effectiveness of KTM-DRL and its knowledge transfer and online learning algorithms, as well as its superiority over the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2020-10-15T03:26:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.