One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

• URL: http://arxiv.org/abs/2608.25936v1
• Date: Wed, 26 Aug 2026 15:52:19 GMT
• Title: One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation
• Abstract summary: On-policy distillation trains a language model on its own generations while a teacher scores them token by token.<n>The teacher is the model itself, conditioned on privileged information the student will not have at test time.<n>Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens.<n>But the same asymmetry that produces the signal also biases it.
• Score: 3.0867286767516937
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.

Related papers

• OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation [15.986808086256792]
  Privileged on-policy distillation improves multimodal reasoning by allowing a teacher to evaluate student trajectories using rich, training-only visual evidence.<n>We find that the teacher's visual corrective preference is not lost under this misleading agreement.<n>We introduce OPD-Aha, which reconstructs the distillation target directly from this isolated visual preference.
  arXiv  Detail & Related papers  (2026-09-15T00:25:28Z)
• Self-Supervised Visual On-Policy Distillation [63.83705343579541]
  Visual on-policy distillation relies heavily on an informative teacher-student asymmetry.<n>We introduce Self-Supervised Visual On-Policy Distillation (S$2$VOPD)<n>S$2$VOPD distills the teacher's distribution conditioned on the original image on-policy into the student distribution conditioned on a strongly augmented view of the same image.
  arXiv  Detail & Related papers  (2026-08-14T09:52:01Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• Geometric Self-Distillation for Reasoning Generalization [12.38444431260744]
  On-policy distillation is a practical post-training recipe for large language models.<n>In privileged-context self-distillation, teacher and student are the same model conditioned on the same prefix.<n>We introduce GeoSD, a geometric self-distillation objective that treats this drift as movement in the student's predictive behavior.
  arXiv  Detail & Related papers  (2026-07-07T23:16:19Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• Not All Synthetic Data Is Yours to Learn From [68.13136636413601]
  We show that weak self-training can amplify capabilities already present in the pretrained model.<n>We study this in the minimal setting of prompt-free unconditional self-training.
  arXiv  Detail & Related papers  (2026-05-29T10:34:11Z)
• Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information [22.436966302243565]
  We propose Anti-Self-Distillation, which ascends a divergence between student and teacher rather than descending it.<n>AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points.
  arXiv  Detail & Related papers  (2026-05-12T06:40:43Z)
• A Survey of On-Policy Distillation for Large Language Models [19.95776080082138]
  This survey provides the first comprehensive overview of On-Policy Distillation for Large Language Models (LLMs)<n>We introduce a unified $f$-divergence framework over on-policy samples and organize the landscape along three dimensions: emphfeedback signal (logit-based, outcome-based, or self-play), emphteacher access (white-box, black-box, or teacher-free), and emphloss minimization (token-level, sequence-level, or hybrid)
  arXiv  Detail & Related papers  (2026-04-01T08:32:34Z)
• Distilling Realizable Students from Unrealizable Teachers [9.968083244726941]
  We study policy distillation under privileged information, where a student policy with only partial observations must learn from a teacher with full-state access.<n>Existing approaches either modify the teacher to produce realizable but sub-optimal demonstrations or rely on the student to explore missing information independently.<n>We introduce two methods: (i) an imitation learning approach that adaptively determines when the student should query the teacher for corrections, and (ii) a reinforcement learning approach that selects where to initialize training for efficient exploration.
  arXiv  Detail & Related papers  (2025-05-14T16:45:51Z)
• Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization [84.86241161706911]
  We show that teacher LLMs can indeed intervene on student reasoning to improve their performance. We also demonstrate that in multi-turn interactions, teacher explanations generalize and learn from explained data. We verify that misaligned teachers can lower student performance to random chance by intentionally misleading them.
  arXiv  Detail & Related papers  (2023-06-15T17:27:20Z)
• Self-Training with Differentiable Teacher [80.62757989797095]
  Self-training achieves enormous success in various semi-supervised and weakly-supervised learning tasks. The method can be interpreted as a teacher-student framework, where the teacher generates pseudo-labels, and the student makes predictions. We propose ours, short for differentiable self-training, that treats teacher-student as a Stackelberg game.
  arXiv  Detail & Related papers  (2021-09-15T02:06:13Z)
• Distilling Double Descent [65.85258126760502]
  Distillation is the technique of training a "student" model based on examples that are labeled by a separate "teacher" model. We show, that, even when the teacher model is highly over parameterized, and provides emphhard labels, using a very large held-out unlabeled dataset can result in a model that outperforms more "traditional" approaches.
  arXiv  Detail & Related papers  (2021-02-13T02:26:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.