Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation

• URL: http://arxiv.org/abs/2608.08176v1
• Date: Sat, 08 Aug 2026 15:13:07 GMT
• Title: Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
• Abstract summary: On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters.<n>We propose Unified On-Policy Self-Distillation (USD), a lightweight online algorithm to solve the Lagrangian.<n>USD consistently demonstrates superior performance over OPSD and token- and PI-side baselines across various model scales.
• Score: 7.759584418408814
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy self-distillation (OPSD) improves the reasoning abilities of LLMs by internalizing privileged context into model parameters through self-distillation. Two recent research lines promote vanilla OPSD by choosing which tokens to learn from and by controlling how much privileged information the teacher receives, respectively. However, we show that each line optimizes one variable while holding the other fixed, which leads to a suboptimal solution. We argue that the two variables are coupled through the student's learning capacity: the privileged information sets the per-token divergence the teacher prescribes, while token weighting selects which of these the student must absorb. We formalize the two lines of work into a unified optimization framework, which maximizes the aggregate teacher--student divergence, subject to a budget on the aggregate learning difficulty the student can absorb. Under this modelling, we propose Unified On-Policy Self-Distillation (USD), a lightweight online algorithm to solve the Lagrangian. USD reveals that a single dual variable governs both decisions: at one price for learning difficulty, it simultaneously sets the token-selection threshold and the direction of privileged-information adjustment, keeping supervision matched to the student's evolving capacity. Through extensive experiments, USD consistently demonstrates superior performance over OPSD and token- and PI-side baselines across various model scales on various reasoning benchmarks. Code is available at https://github.com/lauvlalala/USD.

Related papers

• OPD-V: Visual On-Policy Self-Distillation with Modality Balance [60.972871141106886]
  On-Policy Self-Distillation (OPSD) has become a standard post-training approach for improving visual reasoning in large language models (MLLMs)<n>Existing methods draw privileged information from diverse input sources to guide self-distillation.<n>We introduce OPD-V, a visual OPSD paradigm that instantiates such information through the Positive Teacher and Negative Teacher.
  arXiv  Detail & Related papers  (2026-08-05T17:53:06Z)
• $β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation [85.07190808882523]
  On-policy self-distillation is a promising approach to improve reasoning language models.<n>We show that $$-OPSD consistently outperforms vanilla OPSD.
  arXiv  Detail & Related papers  (2026-07-30T17:41:16Z)
• Weak-to-Strong On-Policy Distillation [21.651421588571296]
  We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a framework that improves the strong student by distilling from multiple weak models.<n>Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker.
  arXiv  Detail & Related papers  (2026-07-28T20:31:48Z)
• Contrastive On-Policy Distillation [36.47487086937924]
  We propose COPD, a contrastive On-policy Distillation framework.<n> COPD directly encourages the student model to learn more concise and efficient reasoning strategies.
  arXiv  Detail & Related papers  (2026-07-21T12:35:04Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe [53.40076304466524]
  We propose Uni-OPD, a unified OPD framework that generalizes across Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs)<n>Specifically, from the student's perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training.<n>We develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories.
  arXiv  Detail & Related papers  (2026-05-05T12:15:21Z)
• AdaSwitch: Adaptive Switching Generation for Knowledge Distillation [58.647880811071495]
  Small language models (SLMs) are crucial for applications with strict latency and computational constraints.<n>We propose AdaSwitch, a novel approach that combines on-policy and off-policy generation at the token level.<n>AdaSwitch consistently improves accuracy, offering a practical and effective method for distilling SLMs with acceptable additional overhead.
  arXiv  Detail & Related papers  (2025-10-09T06:38:37Z)
• A Lagrangian Duality Approach to Active Learning [119.36233726867992]
  We consider the batch active learning problem, where only a subset of the training data is labeled. We formulate the learning problem using constrained optimization, where each constraint bounds the performance of the model on labeled samples. We show, via numerical experiments, that our proposed approach performs similarly to or better than state-of-the-art active learning methods.
  arXiv  Detail & Related papers  (2022-02-08T19:18:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.