Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

• URL: http://arxiv.org/abs/2608.16333v1
• Date: Mon, 17 Aug 2026 09:37:39 GMT
• Title: Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning
• Abstract summary: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories.<n>Standard token-level OPD can provide only fragmented corrections along an erroneous student trajectory.<n>We propose emphStep-Level On-Policy Distillation (SOPD), which combines the long-horizon correction of supervised fine-tuning with the on-policy advantage of OPD.
• Score: 25.14117376141824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conventional off-policy distillation with substantially less data. However, standard token-level OPD can provide only fragmented corrections along an erroneous student trajectory and cannot unfold a complete and correct repair path. Motivated by this limitation, we propose \emph{Step-Level On-Policy Distillation} (SOPD), which combines the long-horizon correction of supervised fine-tuning (SFT) with the on-policy advantage of OPD to provide step-level supervision over complete student-generated trajectories. We show that, at different limits of step length, SOPD reduces to SFT or approximates OPD. Compared with SFT, the teacher responses in SOPD are conditioned on student trajectories and therefore align more closely with student-visited states; compared with OPD, SOPD provides longer-horizon corrections rather than fragmented token-level guidance. Across both reasoning and agent tasks, SOPD substantially outperforms conventional SFT and OPD. For example, on ALFWorld, SOPD improves the average success rate by 13.4 points over Vanilla OPD. We hope this work offers a new perspective for future research on distillation methods.

Related papers

• DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models [11.299607850340607]
  DreOPD is a Degraded-reference extrapolative OPD method for flow-matching models.<n>We show that DreOPD outperforms OPD and multi-task RL baselines in average performance.
  arXiv  Detail & Related papers  (2026-08-10T07:55:57Z)
• Simple-OPD: Demystifying Warm-up for On-policy Distillation [51.693357695326846]
  On-policy distillation (OPD) trains a student on its own rollouts with token-level supervision from teacher models.<n>We demystify warm-up for OPD from both data and training perspectives.
  arXiv  Detail & Related papers  (2026-08-07T04:47:38Z)
• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation [25.25989941933095]
  We propose SAGE-OPD, a verifier-free selective intervention framework specifically designed for multi-turn OPD.<n>Instead of applying teacher supervision uniformly across all turns, SAGE-OPD first observes environment feedback and uses teacher judgment to decide whether each student response should be skipped or intervened on.<n>Experiments on agent tasks show that SAGE-OPD consistently improves over baselines, achieving up to a 13.3% relative improvement in ALFWorld unseen success rate over standard OPD.
  arXiv  Detail & Related papers  (2026-06-17T23:58:14Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• Trajectory-Refined Distillation [25.346810464266497]
  Trajectory-Refined Distillation (TRD) is a trajectory-level correction method that revises the student's rollout under the teacher guidance while within on-policy support.<n> TRD can also be applied to on-policy self-distillation (OPSD), a parameter-sharing variant that uses the student model conditioned on privileged informations as the teacher.
  arXiv  Detail & Related papers  (2026-06-07T03:17:25Z)
• Are Full Rollouts Necessary for On-Policy Distillation? [63.18243901591995]
  On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts.<n>We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency.<n>We propose two simple horizon-control strategies: Progressive OPD, which gradually expands the rollout horizon during training, and Truncated OPD, which permanently performs distillation on reliable truncated rollouts.
  arXiv  Detail & Related papers  (2026-05-29T16:12:54Z)
• SOD: Step-wise On-policy Distillation for Small Language Model Agents [32.49707795291693]
  Tool-integrated reasoning is difficult to scale to small language models.<n>Recently, on-policy distillation has gained popularity by supplying dense token-level supervision from a teacher.<n>We propose SOD, a step-wise on-policy distillation framework for small language model agents.
  arXiv  Detail & Related papers  (2026-05-08T13:30:42Z)
• Rubric-based On-policy Distillation [62.11106822527392]
  On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios.<n>We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses.<n>We introduce ROPD, a simple yet foundational framework for rubric-based OPD. Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization.
  arXiv  Detail & Related papers  (2026-05-08T07:52:15Z)
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
  We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
  arXiv  Detail & Related papers  (2026-04-27T03:38:27Z)
• Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation [7.2992280064983825]
  On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models.<n>Standard OPD requires a live teacher inference server throughout training, resulting in substantial infrastructure overhead.<n>We propose Lightning OPD, an offline on-policy distillation framework that enforces teacher consistency by precomputing teacher log-probabilities over SFT rollouts.
  arXiv  Detail & Related papers  (2026-04-14T17:44:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.