Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

• URL: http://arxiv.org/abs/2605.10889v1
• Date: Mon, 11 May 2026 17:33:28 GMT
• Title: Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
• Abstract summary: We introduce a training-free diagnostic framework that operates at the highest resolution: per token, per question, and per teacher.<n>We observe that distillation guidance exhibits substantially higher alignment with the ideal on incorrect rollouts than on correct ones.
• Score: 32.00194810443734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy distillation offers dense, per-token supervision for training reasoning models; however, it remains unclear under which conditions this signal is beneficial and under which it is detrimental. Which teacher model should be used, and in the case of self-distillation, which specific context should serve as the supervisory signal? Does the optimal choice vary from one token to the next? At present, addressing these questions typically requires costly training runs whose aggregate performance metrics obscure the dynamics at the level of individual tokens. We introduce a training-free diagnostic framework that operates at the highest resolution: per token, per question, and per teacher. We derive an ideal per-node gradient defined as the parameter update that maximally increases the student's probability of success. We then develop a scalable targeted-rollout algorithm to estimate this gradient efficiently, even for long chains of intermediate thoughts. The gradient alignment score, defined as the cosine similarity between this ideal gradient and any given distillation gradient, quantifies the extent to which a particular configuration approximates the ideal signal. Across a range of self-distillation settings and external teacher models, we observe that distillation guidance exhibits substantially higher alignment with the ideal on incorrect rollouts than on correct ones, where the student already performs well and the teacher's signal tends to become noisy. Furthermore, we find that the optimal distillation context depends jointly on the student model's capacity and the target task, and that no single universally effective configuration emerges. These findings motivate the use of per-task, per-token diagnostic analyses for distillation.

Related papers

• Beyond Trajectory Matching: Reflow with Marginal Distribution Alignment [7.13416637929041]
  Diffusion and continuous-flow generative models achieve high-quality generation.<n>Reflow-based distillation simplifies teacher ODE trajectories so that a student model can approximate the teacher transport with fewer steps.<n>We introduce a marginal-alignment regularizer that penalizes the discrepancy between the student-induced marginal and the corresponding teacher marginal at the endpoint of each distillation interval.
  arXiv  Detail & Related papers  (2026-06-28T09:19:44Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation [49.98710755440242]
  Distillation enables compact Vision-Language Models (VLMs) to obtain strong reasoning capabilities.<n>We reveal a critical inefficiency in this approach: up to 69% of the prompts in standard chart / document reasoning datasets are effectively zero-delta.<n>We propose a staged synthesis pipeline that repurposes existing datasets as seeds, actively targeting student failure modes to produce better prompts.
  arXiv  Detail & Related papers  (2026-05-15T02:04:12Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• Positive-Unlabeled Reinforcement Learning Distillation for On-Premise Small Models [130.8912476550625]
  We propose a positive-unlabeled (PU) reinforcement learning distillation method for on-premise small-model deployment.<n>Our method distills the teacher's preference-optimization capability from black-box generations into a locally trainable student.<n>Experiments demonstrate that our method achieves consistently strong performance under a low-cost setting.
  arXiv  Detail & Related papers  (2026-01-28T15:14:50Z)
• Long-Chain Reasoning Distillation via Adaptive Prefix Alignment [57.130176131042965]
  We propose a framework that exploits teacher CoTs for distillation through adaptive prefix alignment.<n>P-ALIGN adaptively truncates teacher-generated reasoning trajectories by determining whether the remaining suffix is concise.<n>Experiments on multiple mathematical reasoning benchmarks demonstrate that P-ALIGN outperforms all baselines by over 3%.
  arXiv  Detail & Related papers  (2026-01-15T04:40:45Z)
• Elucidating the Preconditioning in Consistency Distillation [25.213664260896103]
  We propose a principled way dubbed textitAnalytic-Precond to analytically optimize the preconditioning according to the consistency gap.<n>We demonstrate that Analytic-Precond can facilitate the learning of trajectory jumpers, enhance the alignment of the student trajectory with the teacher's, and achieve $2times$ to $3times$ training acceleration of consistency trajectory models.
  arXiv  Detail & Related papers  (2025-02-05T06:30:37Z)
• Logit Standardization in Knowledge Distillation [83.31794439964033]
  The assumption of a shared temperature between teacher and student implies a mandatory exact match between their logits in terms of logit range and variance. We propose setting the temperature as the weighted standard deviation of logit and performing a plug-and-play Z-score pre-process of logit standardization. Our pre-process enables student to focus on essential logit relations from teacher rather than requiring a magnitude match, and can improve the performance of existing logit-based distillation methods.
  arXiv  Detail & Related papers  (2024-03-03T07:54:03Z)
• How a student becomes a teacher: learning and forgetting through Spectral methods [1.1470070927586018]
  In theoretical ML, the teacher paradigm is often employed as an effective metaphor for real-life tuition. In this work, we take a leap forward by proposing a radically different optimization scheme. Working in this framework, we could isolate a stable student substructure, that mirrors the true complexity of the teacher.
  arXiv  Detail & Related papers  (2023-10-19T09:40:30Z)
• Knowledge Diffusion for Distillation [53.908314960324915]
  The representation gap between teacher and student is an emerging topic in knowledge distillation (KD) We state that the essence of these methods is to discard the noisy information and distill the valuable information in the feature. We propose a novel KD method dubbed DiffKD, to explicitly denoise and match features using diffusion models.
  arXiv  Detail & Related papers  (2023-05-25T04:49:34Z)
• HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers [49.79405257763856]
  This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. We propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning.
  arXiv  Detail & Related papers  (2023-02-19T17:37:24Z)
• Paced-Curriculum Distillation with Prediction and Label Uncertainty for Image Segmentation [25.20877071896899]
  In curriculum learning, the idea is to train on easier samples first and gradually increase the difficulty. In self-paced learning, a pacing function defines the speed to adapt the training progress. We develop a novel paced-curriculum distillation (PCD) for medical image segmentation.
  arXiv  Detail & Related papers  (2023-02-02T12:24:14Z)
• Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance Segmentation [54.49894381464853]
  We propose to leverage both labeled and unlabeled data for instance segmentation with improved accuracy by knowledge distillation. We propose a novel Mask-guided Mean Teacher framework with Perturbation-sensitive Sample Mining. Experiments show that the proposed method improves the performance significantly compared with the supervised method learned from labeled data only.
  arXiv  Detail & Related papers  (2020-07-21T13:27:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.