Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

• URL: http://arxiv.org/abs/2604.13016v2
• Date: Wed, 15 Apr 2026 17:48:51 GMT
• Title: Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
• Abstract summary: On-policy distillation (OPD) has become a core technique in the post-training of large language models.<n>This paper provides a systematic investigation of OPD dynamics and mechanisms.
• Score: 36.507858889603845
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.

Related papers

• SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation [28.886137823815826]
  We introduce Sparse Probing and Outcome-calibrated Targets OPD (SPOT)<n>SPOT addresses two coupled decisions, where to probe and what to distill, through an acquisition--exploration-exploitation procedure.<n>Experiments across multiple student models and reasoning benchmarks demonstrate the effectiveness of SPOT.
  arXiv  Detail & Related papers  (2026-08-05T03:59:24Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• DOPD: Dual On-policy Distillation [59.65986569617147]
  On-policy distillation offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals.<n>We propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies.<n>Experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD.
  arXiv  Detail & Related papers  (2026-06-29T17:55:53Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• Adversarial Dual On-Policy Distillation from Expressive Teacher [53.63143943452487]
  Flow Matching teacher is learned from demonstrations and co-trained with a lightweight student.<n>The teacher provides two complementary signals on student rollouts.<n>The action channel supplies dense local targets at student-visited states, stabilizing exploitation.
  arXiv  Detail & Related papers  (2026-05-26T14:38:03Z)
• Less is More: Early Stopping Rollout for On-Policy Distillation [39.392596318514244]
  Early Stopping Rollout (ESR) is a simple yet effective distillation strategy that simply restricts the rollout generation to the first response tokens.<n>We show that ESR both surpasses the full rollout OPD performance across model size, family, tasks and training regime.
  arXiv  Detail & Related papers  (2026-05-26T13:49:37Z)
• SOD: Step-wise On-policy Distillation for Small Language Model Agents [32.49707795291693]
  Tool-integrated reasoning is difficult to scale to small language models.<n>Recently, on-policy distillation has gained popularity by supplying dense token-level supervision from a teacher.<n>We propose SOD, a step-wise on-policy distillation framework for small language model agents.
  arXiv  Detail & Related papers  (2026-05-08T13:30:42Z)
• Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe [53.40076304466524]
  We propose Uni-OPD, a unified OPD framework that generalizes across Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs)<n>Specifically, from the student's perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training.<n>We develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories.
  arXiv  Detail & Related papers  (2026-05-05T12:15:21Z)
• SODA: Semi On-Policy Black-Box Distillation for Large Language Models [19.748920924563546]
  Black-box knowledge distillation for large language models presents a strict trade-off.<n>Semi evaluations across four compact Qwen2.5 and Llama-3 models validate this semi on-policy paradigm.<n>It achieves this superior distillation quality while training 10 times faster, consuming 27% less peak GPU memory, and completely eliminating adversarial instability.
  arXiv  Detail & Related papers  (2026-04-04T21:38:22Z)
• Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation [57.524909883706556]
  On-policy distillation (OPD) has demonstrated strong empirical gains in improving student performance.<n>This work introduces a flexible reference model and a reward scaling factor that controls the relative weight of the reward term against the KL regularization.<n>In particular, in the setting where we merge the knowledge from different domain experts, obtained by applying domain-specific RL to the same student model, ExOPD enables the student to even surpass the teacher's performance boundary.
  arXiv  Detail & Related papers  (2026-02-12T16:14:29Z)
• AdaSwitch: Adaptive Switching Generation for Knowledge Distillation [58.647880811071495]
  Small language models (SLMs) are crucial for applications with strict latency and computational constraints.<n>We propose AdaSwitch, a novel approach that combines on-policy and off-policy generation at the token level.<n>AdaSwitch consistently improves accuracy, offering a practical and effective method for distilling SLMs with acceptable additional overhead.
  arXiv  Detail & Related papers  (2025-10-09T06:38:37Z)
• Distilling Realizable Students from Unrealizable Teachers [9.968083244726941]
  We study policy distillation under privileged information, where a student policy with only partial observations must learn from a teacher with full-state access.<n>Existing approaches either modify the teacher to produce realizable but sub-optimal demonstrations or rely on the student to explore missing information independently.<n>We introduce two methods: (i) an imitation learning approach that adaptively determines when the student should query the teacher for corrections, and (ii) a reinforcement learning approach that selects where to initialize training for efficient exploration.
  arXiv  Detail & Related papers  (2025-05-14T16:45:51Z)
• Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones? [58.80794196076336]
  Distilling large language models (LLMs) typically involves transferring the teacher model's responses through supervised fine-tuning (SFT)<n>We propose a novel distillation pipeline that transfers both responses and rewards.<n>Our method generates pseudo-rewards through a self-supervised mechanism that leverages the inherent structure of both teacher and student responses.
  arXiv  Detail & Related papers  (2025-02-26T20:50:11Z)
• Self-Evolution Knowledge Distillation for LLM-based Machine Translation [36.01859033056453]
  We propose a distillation strategy called Self-Evolution KD.<n>The core of this approach involves dynamically integrating teacher distribution and one-hot distribution of ground truth into the student distribution as prior knowledge.<n> Experimental results show our method brings an average improvement of approximately 1.4 SacreBLEU points across four translation directions in the WMT22 test sets.
  arXiv  Detail & Related papers  (2024-12-19T12:24:15Z)
• Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step [133.60124577507727]
  Chain-of-thought prompting primes large language models to verbalize rationalization for their predictions. We show that orders-of-magnitude smaller models can still benefit from chain-of-thought prompting. We introduce Chain-of-Thought Distillation (SCoTD), a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model.
  arXiv  Detail & Related papers  (2023-06-24T20:15:07Z)
• On student-teacher deviations in distillation: does it pay to disobey? [54.908344098305804]
  Knowledge distillation has been widely used to improve the test accuracy of a "student" network. Despite being trained to fit the teacher's probabilities, the student may not only significantly deviate from the teacher probabilities, but may also outdo the teacher in performance.
  arXiv  Detail & Related papers  (2023-01-30T14:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.