Interactive Distillation for Cooperative Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2601.05407v1
• Date: Thu, 08 Jan 2026 22:16:43 GMT
• Title: Interactive Distillation for Cooperative Multi-Agent Reinforcement Learning
• Abstract summary: HINT (Hierarchical INteractive Teacher-based transfer) is a novel KD framework for MARL in a centralized training, decentralized execution setup.<n>By leveraging hierarchical RL, HINT provides a scalable, high-performing teacher.<n>We evaluate HINT on challenging cooperative domains (e.g., FireCommander for resource allocation, MARINE for tactical combat)
• Score: 0.49703640686206074
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Knowledge distillation (KD) has the potential to accelerate MARL by employing a centralized teacher for decentralized students but faces key bottlenecks. Specifically, there are (1) challenges in synthesizing high-performing teaching policies in complex domains, (2) difficulties when teachers must reason in out-of-distribution (OOD) states, and (3) mismatches between the decentralized students' and the centralized teacher's observation spaces. To address these limitations, we propose HINT (Hierarchical INteractive Teacher-based transfer), a novel KD framework for MARL in a centralized training, decentralized execution setup. By leveraging hierarchical RL, HINT provides a scalable, high-performing teacher. Our key innovation, pseudo off-policy RL, enables the teacher policy to be updated using both teacher and student experience, thereby improving OOD adaptation. HINT also applies performance-based filtering to retain only outcome-relevant guidance, reducing observation mismatches. We evaluate HINT on challenging cooperative domains (e.g., FireCommander for resource allocation, MARINE for tactical combat). Across these benchmarks, HINT outperforms baselines, achieving improvements of 60% to 165% in success rate.

Related papers

• From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents [16.67357658768852]
  We propose Outcome-Verified Comparative Self-Distillation (OVCSD)<n>OVCSD organizes failed student rollouts into a prefix tree, adaptively invokes a skill-conditioned teacher from student-reached states, and retains only outcome-verified successful continuations.<n> Experiments on ALFWorld and WebShop across three model scales show that OVCSD consistently outperforms skill-free RL and existing self-distillation baselines.
  arXiv  Detail & Related papers  (2026-07-30T09:47:58Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• Teacher-Guided Policy Optimization for LLM Distillation [90.49982387646861]
  Teacher-Guided Policy Optimization (TGPO) is an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout.<n>Experiments on complex reasoning benchmarks demonstrate that TGPO significantly outperforms standard baselines and is robust to different teachers.
  arXiv  Detail & Related papers  (2026-05-13T09:20:03Z)
• Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents [24.99615788156812]
  We introduce Skill-SD, a framework that turns the agent's own trajectories into dynamic training-only supervision.<n>We derive an importance-weighted reverse-KL loss to provide gradient-correct token-level distillation.<n> Experimental results on agentic benchmarks demonstrate that Skill-SD substantially outperforms the standard RL baseline.
  arXiv  Detail & Related papers  (2026-04-12T14:57:52Z)
• Reinforcement-aware Knowledge Distillation for LLM Reasoning [63.53679456364683]
  Reinforcement learning (RL) post-training has recently driven gains in long chain-of-thought reasoning large language models (LLMs)<n>Most existing knowledge distillation methods are designed for supervised fine-tuning (SFT), relying on fixed teacher traces or teacher-student Kullback-Leibler (KL) divergence-based regularization.<n>We propose RL-aware distillation (RLAD), which performs selective imitation during RL -- guiding the student toward the teacher only when it improves the current policy update.
  arXiv  Detail & Related papers  (2026-02-26T00:20:39Z)
• Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation [57.524909883706556]
  On-policy distillation (OPD) has demonstrated strong empirical gains in improving student performance.<n>This work introduces a flexible reference model and a reward scaling factor that controls the relative weight of the reward term against the KL regularization.<n>In particular, in the setting where we merge the knowledge from different domain experts, obtained by applying domain-specific RL to the same student model, ExOPD enables the student to even surpass the teacher's performance boundary.
  arXiv  Detail & Related papers  (2026-02-12T16:14:29Z)
• Conservative Bias in Multi-Teacher Learning: Why Agents Prefer Low-Reward Advisors [0.0]
  This paper reveals an unexpected phenomenon in interactive reinforcement learning (IRL)<n>When given a choice between teachers with different reward structures, learning agents overwhelmingly prefer conservative, low-reward teachers.
  arXiv  Detail & Related papers  (2025-12-19T02:38:04Z)
• AdaSwitch: Adaptive Switching Generation for Knowledge Distillation [58.647880811071495]
  Small language models (SLMs) are crucial for applications with strict latency and computational constraints.<n>We propose AdaSwitch, a novel approach that combines on-policy and off-policy generation at the token level.<n>AdaSwitch consistently improves accuracy, offering a practical and effective method for distilling SLMs with acceptable additional overhead.
  arXiv  Detail & Related papers  (2025-10-09T06:38:37Z)
• Learning from Diverse Reasoning Paths with Routing and Collaboration [65.77515749498575]
  We propose Quality-filtered Routing with Cooperative Distillation (QR-Distill), combining path quality filtering, conditional routing, and peer teaching.<n> Experiments demonstrate QR-Distill's superiority over traditional single- and multi-path distillation methods.
  arXiv  Detail & Related papers  (2025-08-23T01:15:57Z)
• TGRL: An Algorithm for Teacher Guided Reinforcement Learning [45.38447023752256]
  It is common to train a policy to maximize a combination of reinforcement and teacher-student learning objectives. We present a $textitprincipled$ approach, along with an approximate implementation for $textitdynamically$ and $textitautomatically$ balancing when to follow the teacher and when to use rewards.
  arXiv  Detail & Related papers  (2023-07-06T17:58:40Z)
• Guarded Policy Optimization with Imperfect Online Demonstrations [32.22880650876471]
  Teacher-Student Framework is a reinforcement learning setting where a teacher agent guards the training of a student agent. It is expensive or even impossible to obtain a well-performing teacher policy. We develop a new method that can incorporate arbitrary teacher policies with modest or inferior performance.
  arXiv  Detail & Related papers  (2023-03-03T06:24:04Z)
• Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation [70.92135839545314]
  We propose the dynamic prior knowledge (DPK), which integrates part of teacher's features as the prior knowledge before the feature distillation. Our DPK makes the performance of the student model positively correlated with that of the teacher model, which means that we can further boost the accuracy of students by applying larger teachers.
  arXiv  Detail & Related papers  (2022-06-13T11:52:13Z)
• CTDS: Centralized Teacher with Decentralized Student for Multi-Agent Reinforcement Learning [114.69155066932046]
  This work proposes a novel. Teacher with Decentralized Student (C TDS) framework, which consists of a teacher model and a student model. Specifically, the teacher model allocates the team reward by learning individual Q-values conditioned on global observation. The student model utilizes the partial observations to approximate the Q-values estimated by the teacher model.
  arXiv  Detail & Related papers  (2022-03-16T06:03:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.