DualOPSD: Adaptive Privileged Teachers for On-Policy Self-Distillation

• URL: http://arxiv.org/abs/2608.26019v1
• Date: Wed, 26 Aug 2026 17:01:21 GMT
• Title: DualOPSD: Adaptive Privileged Teachers for On-Policy Self-Distillation
• Abstract summary: On-policy self-distillation (OPSD) uses a privileged copy of the student model to provide dense supervision without an external teacher.<n>We propose DualOPSD, an asymmetric alternating framework that adapts both policies.
• Score: 11.771522030770228
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy self-distillation (OPSD) uses a privileged copy of the student model to provide dense supervision without an external teacher. OPSD keeps this privileged teacher fixed, even though the student distribution and output style change during training. We propose DualOPSD, an asymmetric alternating framework that adapts both policies. The student first learns from the privileged teacher. The teacher then moves toward the updated student distribution on the same student trajectory. This update makes later supervision responsive to the learner and does not require another rollout. On Qwen3-8B in non-thinking mode, DualOPSD improves avg@12 over OPSD by 23.61, 13.89, and 10.00 points on AIME 2024, AIME 2025, and HMMT 2025. Results at 1.7B and 4B show that the accuracy gain depends on model scale. Across all three scales, DualOPSD reduces truncation. The 4B diagnostic also shows lower KL in both directions between the teacher and student.

Related papers

• VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation [19.198000863807316]
  On-policy self-distillation improves reasoning by training a problem-only student on its own rollouts using dense token-level supervision from a privileged teacher.<n>Standard OPSD treats the teacher distribution as a fixed target along the student's rollout and updates only the student %, although privileged conditioning does not guarantee that the teacher always provides the most appropriate target for problem-only reasoning.<n>We introduce Verifier-Informed Student-to-Teacher Adaptation (VISTA), which preserves the standard OPSD student update while using outcome-verified rollouts to adapt the teacher toward the student distribution.
  arXiv  Detail & Related papers  (2026-08-28T13:10:34Z)
• When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation [21.86522736658343]
  We propose Reward-Aligned On-Policy Distillation (RA-OPD) to mitigate misaligned teacher guidance.<n> RA-OPD selects more reliable trajectories to improve student model performance without requiring additional computational cost.<n>We evaluate RA-OPD on math and code benchmarks using models from the Qwen3 family and the DeepSeek-R1 family.
  arXiv  Detail & Related papers  (2026-08-28T06:05:50Z)
• PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation [1.1440940588722308]
  On-policy self-distillation (OPSD) uses a privileged teacher to supervise a reasoning model on prefixes sampled from its own rollouts.<n>We introduce Privileged Adaptation from Student Trajectories (PAST)<n>PAST treats each completed student trajectory as additional privileged information for the OPSD teacher while leaving the student's distillation prefixes unchanged.
  arXiv  Detail & Related papers  (2026-08-09T14:20:12Z)
• Visual Contrastive Self-Distillation [61.298159957622886]
  We propose Visual Contrastive Self-Distillation, which converts image-content removal into an on-policy self-distillation signal.<n>We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student.
  arXiv  Detail & Related papers  (2026-07-23T17:37:37Z)
• OPOD: On-Policy Omni Distillation [55.440058537827134]
  Training a single model on pooled multimodal data often fails to match models specialized for individual modalities.<n>We present On-Policy Omni Distillation (OPD), which routes each student response to the matching text, image, or audio teacher.<n>OPD achieves the best average score at every scale, reaching 70.8, 51.7, and 46.2.
  arXiv  Detail & Related papers  (2026-07-23T04:55:04Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• DemoPSD: Disagreement-Modulated Policy Self-Distillation [49.30809363804017]
  On-policy self-distillation has emerged as a practical method for training large language models.<n>DemoPSD steers the student toward a weighted geometric combination of the teacher and student distributions.
  arXiv  Detail & Related papers  (2026-07-02T17:58:29Z)
• Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients [89.89504265663057]
  Zone of Proximal Policy Optimization (ZPPO) is inspired by Vygotsky's zone of proximal development.<n>ZPPO pairs one correct teacher response with one incorrect student response as anonymized candidates the student must discriminate.<n>A prompt replay buffer recirculates each hard question until it either graduates-the student's mean rollout accuracy on it reaches half- or is FIFO-evicted.
  arXiv  Detail & Related papers  (2026-06-16T17:46:02Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning [45.79647925282674]
  On-policy self-distillation (OPSD) trains a student on its own rollouts using a privileged teacher.<n>Existing entropy-based OPD methods relax this uniformity by modulating token-level supervision with teacher entropy.<n>We propose Position-Weighted On-Policy Self-Distillation (PW-OPSD), which applies an increasing position weight while keeping the same student rollout, privileged teacher pass, and clipped forward-KL target as OPSD.
  arXiv  Detail & Related papers  (2026-05-20T18:14:03Z)
• Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning [2.9029480541171107]
  We propose adaptive teacher exposure as an effective new axis for reasoning self-distillation.<n>We show that Adaptive Teacher Exposure for Self-Distillation (ATESD) consistently outperforms competitive self-distillation and RL baselines.
  arXiv  Detail & Related papers  (2026-05-12T03:15:58Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.