Rethinking Privileged Information in On-Policy Self-Distillation

• URL: http://arxiv.org/abs/2608.18271v1
• Date: Tue, 18 Aug 2026 19:42:16 GMT
• Title: Rethinking Privileged Information in On-Policy Self-Distillation
• Abstract summary: On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information.<n>We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model.
• Score: 45.88028371034407
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information. We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model. We perform OPSD experiments on science and mathematics datasets using Qwen3 models ranging from 1.7B to 8B. Our analysis framework separates the supervision induced by the reference from the supervision provided by the teacher without the reference and measures how each aligns with changes in the student's predictions. The correct reference does not provide a consistent performance benefit across teacher generation modes, model sizes, and training datasets. Students can improve without the correct reference, and a solution from another problem can outperform the correct solution on several mathematical reasoning benchmarks. The student's predictions align more strongly with the base model's thinking behavior than with the supervision induced by the reference, but controls constructed from other problems reproduce much of both alignments. Moreover, stronger alignment attributable to the correct reference does not reliably coincide with a greater performance benefit from the reference. Performance gains and distributional alignment alone therefore cannot determine how privileged reference information contributes to student learning in OPSD.

Related papers

• ADOPD: Reference-Privileged On-Policy Distillation for MLLM-Based Industrial Anomaly Detection [15.5664235470661]
  Industrial anomaly detection requires identifying fine-grained deviations from normal visual patterns.<n>Access to references during training allows a reference-aware teacher to supervise a query-only student.<n>We propose ADOPD, a reference-privileged on-policy distillation framework.
  arXiv  Detail & Related papers  (2026-08-10T16:12:39Z)
• Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning [17.645367206348222]
  On-policy self-distillation improves reasoning by using a privileged view of a model conditioned on reference solutions to supervise a student view that observes only the question.<n>We propose Problem-Space-Guided OPSD (PS-OPSD), which replaces the complete solution with trajectory-grounded guidance describing the initial state, goal conditions, constraints, and a selected state-transition path.
  arXiv  Detail & Related papers  (2026-08-03T01:47:25Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• Purified OPSD: On-Policy Self-Distillation Without Losing How to Think [52.879387744920415]
  We find that on-policy self-distillation (OPSD) consistently fails on long chain-of-thought (long-CoT) reasoning models.<n>We propose a two-step solution: first, we construct a reference-only teacher to isolate the non-transferable component of the supervision signal.<n>Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution.
  arXiv  Detail & Related papers  (2026-07-02T14:33:07Z)
• Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teacher [18.162030833167385]
  Chest X-ray visual question answering (CXR VQA) requires models not only to predict correct answers, but also to produce reliable medical reasoning.<n>We propose text-to-Reason (T2R), a framework that introduces comparison-based supervision into CoT optimization.
  arXiv  Detail & Related papers  (2026-06-24T05:07:41Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals [65.71225905458182]
  Self-distillation enables language models to learn on-policy from their own trajectories by using the same model as both student and teacher.<n>This setup provides dense token-level feedback without relying on a separate external model.<n>We introduce AVSD, a novel method of self-distillation with multiple privileged-information views.
  arXiv  Detail & Related papers  (2026-05-20T03:06:36Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• Teacher-Guided Policy Optimization for LLM Distillation [90.49982387646861]
  Teacher-Guided Policy Optimization (TGPO) is an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout.<n>Experiments on complex reasoning benchmarks demonstrate that TGPO significantly outperforms standard baselines and is robust to different teachers.
  arXiv  Detail & Related papers  (2026-05-13T09:20:03Z)
• OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning [19.98950359294245]
  We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectories.<n>Despite the performance gains of OPSD, we identify a common but often overlooked mismatch between teacher and student responses.
  arXiv  Detail & Related papers  (2026-05-12T17:00:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.