Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation

• URL: http://arxiv.org/abs/2608.09228v1
• Date: Mon, 10 Aug 2026 07:51:45 GMT
• Title: Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation
• Abstract summary: On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information.<n>We investigate the role of target-specific privilege with $mathrmOP2mathrmSD$.
• Score: 6.053609157192373
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with $\mathrm{OP}^{2}\mathrm{SD}$ (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, $\mathrm{OP}^{2}\mathrm{SD}$ improves over the base model, remains competitive with OPSD. The success of $\mathrm{OP}^{2}\mathrm{SD}$ implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.

Related papers

• Rethinking Privileged Information in On-Policy Self-Distillation [45.88028371034407]
  On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information.<n>We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model.
  arXiv  Detail & Related papers  (2026-08-18T19:42:16Z)
• Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation [7.759584418408814]
  On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters.<n>We propose Unified On-Policy Self-Distillation (USD), a lightweight online algorithm to solve the Lagrangian.<n>USD consistently demonstrates superior performance over OPSD and token- and PI-side baselines across various model scales.
  arXiv  Detail & Related papers  (2026-08-08T15:13:07Z)
• Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning [17.645367206348222]
  On-policy self-distillation improves reasoning by using a privileged view of a model conditioned on reference solutions to supervise a student view that observes only the question.<n>We propose Problem-Space-Guided OPSD (PS-OPSD), which replaces the complete solution with trajectory-grounded guidance describing the initial state, goal conditions, constraints, and a selected state-transition path.
  arXiv  Detail & Related papers  (2026-08-03T01:47:25Z)
• $β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation [85.07190808882523]
  On-policy self-distillation is a promising approach to improve reasoning language models.<n>We show that $$-OPSD consistently outperforms vanilla OPSD.
  arXiv  Detail & Related papers  (2026-07-30T17:41:16Z)
• Weak-to-Strong On-Policy Distillation [21.651421588571296]
  We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a framework that improves the strong student by distilling from multiple weak models.<n>Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker.
  arXiv  Detail & Related papers  (2026-07-28T20:31:48Z)
• Visual Contrastive Self-Distillation [61.298159957622886]
  We propose Visual Contrastive Self-Distillation, which converts image-content removal into an on-policy self-distillation signal.<n>We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student.
  arXiv  Detail & Related papers  (2026-07-23T17:37:37Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• SocraticPO: Policy Optimization via Interactive Guidance [64.7524628699057]
  Reinforcement learning (RL) for large language models usually supervises reasoning with scalar outcome rewards, such as binary correctness.<n>We propose textbfSocraticPO (Socratic Policy Optimization), a policy-optimization framework that augments RL rollouts with Socratic-style natural-language guidance.
  arXiv  Detail & Related papers  (2026-06-03T09:08:29Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.