Adversarial Dual On-Policy Distillation from Expressive Teacher

• URL: http://arxiv.org/abs/2605.27095v2
• Date: Mon, 01 Jun 2026 11:46:42 GMT
• Title: Adversarial Dual On-Policy Distillation from Expressive Teacher
• Abstract summary: Flow Matching teacher is learned from demonstrations and co-trained with a lightweight student.<n>The teacher provides two complementary signals on student rollouts.<n>The action channel supplies dense local targets at student-visited states, stabilizing exploitation.
• Score: 53.63143943452487
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning from demonstrations in embodied control is often cast as behavioral cloning, and recent diffusion or flow-matching policies improve this paradigm by modeling multi-modal expert actions. Yet these methods remain offline supervised learners: the policy is trained only on expert states and receives no corrective signal on the states it actually visits. On-policy distillation (OPD) offers a natural remedy, but standard OPD assumes a strong fixed teacher, which is unavailable in demonstration-only control. We propose \textbf{FA-OPD}, an \emph{adversarial dual on-policy distillation} method in which a Flow Matching (FM) teacher is learned from demonstrations and co-trained with a lightweight MLP student. The teacher provides two complementary signals on student rollouts. The reward channel learns an expert-likeness objective over state-action pairs and drives online exploration through long-horizon policy optimization. The action channel supplies dense local targets at student-visited states, stabilizing exploitation. FA-OPD couples them so that reward distillation enables generalization beyond point-wise demonstrations, while action distillation keeps exploration anchored near expert-like behavior. Across six robot navigation, manipulation, and locomotion benchmarks, FA-OPD beats strong baselines and shows much stronger robustness under noisy or limited demonstrations. Source code: https://github.com/vanzll/FA-OPD.

Related papers

• On-Policy Delta Distillation [61.76889440384448]
  On-policy distillation is an alternative post-training method in reinforcement learning.<n>We introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution.<n>We show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$2$)
  arXiv  Detail & Related papers  (2026-07-16T16:07:19Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• DOPD: Dual On-policy Distillation [59.65986569617147]
  On-policy distillation offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals.<n>We propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies.<n>Experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD.
  arXiv  Detail & Related papers  (2026-06-29T17:55:53Z)
• On-Policy Adversarial Flow Distillation for Autoregressive Video Generation [34.70887671905755]
  Adversarial Flow Distillation (AFD) is an on-policy framework for heterogeneous black-box video distillation.<n>AFD queries the teacher and rolls out the current student on the same prompts.<n>AFD provides dense velocity-field supervision while requiring no teacher scores, latents, denoising trajectories, step alignment, or reverse-chain reinforcement learning.
  arXiv  Detail & Related papers  (2026-05-25T17:58:12Z)
• Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation [43.020181623750865]
  We propose Monitoring Trajectories and Backtracking when it strays (MOTAB)<n>MOTAB monitors the student's on-policy generation against an adaptive safety boundary.<n>Experiments on the LIMO-v2 and AceReason datasets demonstrate that MOTAB effectively alleviates the dual exposure biases.
  arXiv  Detail & Related papers  (2026-05-19T06:42:52Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• Multi-Rollout On-Policy Distillation via Peer Successes and Failures [31.731266689209914]
  Large language models are often post-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails.<n>On-policy distillation (OPD) offers denser token-level supervision by training on student-generated trajectories.<n>We introduce Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that uses the student's local rollout group to construct more informative teacher signals.
  arXiv  Detail & Related papers  (2026-05-12T18:57:44Z)
• Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe [53.40076304466524]
  We propose Uni-OPD, a unified OPD framework that generalizes across Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs)<n>Specifically, from the student's perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training.<n>We develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories.
  arXiv  Detail & Related papers  (2026-05-05T12:15:21Z)
• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe [36.507858889603845]
  On-policy distillation (OPD) has become a core technique in the post-training of large language models.<n>This paper provides a systematic investigation of OPD dynamics and mechanisms.
  arXiv  Detail & Related papers  (2026-04-14T17:54:28Z)
• AdaSwitch: Adaptive Switching Generation for Knowledge Distillation [58.647880811071495]
  Small language models (SLMs) are crucial for applications with strict latency and computational constraints.<n>We propose AdaSwitch, a novel approach that combines on-policy and off-policy generation at the token level.<n>AdaSwitch consistently improves accuracy, offering a practical and effective method for distilling SLMs with acceptable additional overhead.
  arXiv  Detail & Related papers  (2025-10-09T06:38:37Z)
• Adversarial Prompt Distillation for Vision-Language Models [61.39214202062028]
  Adversarial Prompt Tuning (APT) applies adversarial training during the process of prompt tuning.<n>APD is a bimodal knowledge distillation framework that enhances APT by integrating it with multi-modal knowledge transfer.<n>Extensive experiments on multiple benchmark datasets demonstrate the superiority of our APD method over the current state-of-the-art APT methods.
  arXiv  Detail & Related papers  (2024-11-22T03:02:13Z)
• Beyond Joint Demonstrations: Personalized Expert Guidance for Efficient Multi-Agent Reinforcement Learning [54.40927310957792]
  We introduce a novel concept of personalized expert demonstrations, tailored for each individual agent or, more broadly, each individual type of agent within a heterogeneous team.<n>These demonstrations solely pertain to single-agent behaviors and how each agent can achieve personal goals without encompassing any cooperative elements.<n>We propose an approach that selectively utilizes personalized expert demonstrations as guidance and allows agents to learn to cooperate.
  arXiv  Detail & Related papers  (2024-03-13T20:11:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.