Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning

• URL: http://arxiv.org/abs/2605.11458v1
• Date: Tue, 12 May 2026 03:15:58 GMT
• Title: Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning
• Abstract summary: We propose adaptive teacher exposure as an effective new axis for reasoning self-distillation.<n>We show that Adaptive Teacher Exposure for Self-Distillation (ATESD) consistently outperforms competitive self-distillation and RL baselines.
• Score: 2.9029480541171107
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-policy self-distillation has become a strong recipe for LLM reasoning, where a privileged teacher supervises the student's own rollouts while conditioning on the reference solution. A design choice shared by nearly all such methods, however, has gone unquestioned: the teacher always sees the full reference reasoning. We argue that this default itself is part of the problem and identify a teacher-side exposure mismatch: when the teacher conditions on reasoning far beyond the student's current competence, the resulting token targets become too strong to absorb. A controlled fixed-exposure sweep makes this concrete on two fronts: 1) full exposure is not reliably the best choice, and 2) student-teacher mismatch grows monotonically as the teacher sees more privileged reasoning. This motivates treating teacher exposure not as a fixed hyperparameter but as a learnable training-time control variable. We therefore propose Adaptive Teacher Exposure for Self-Distillation (ATESD). ATESD models the reveal ratio with a lightweight Beta-policy controller conditioned on compact training-state statistics, and uses one sampled exposure for a short hold window of student updates. To make this exposure controller learnable, we optimize it with a discounted learning-progress reward that scores each held decision by its effect on the student's future improvement rather than its immediate loss change, addressing the delayed credit assignment induced by on-policy distillation. Experiments on AIME 24, AIME 25, and HMMT 25 across Qwen3-{1.7B, 4B, 8B} show that ATESD consistently outperforms competitive self-distillation and RL baselines, improving over OPSD by +0.95, +2.05, and +2.33 Average@12 points respectively, and establishing adaptive teacher exposure as an effective new axis for reasoning self-distillation.

Related papers

• Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback [24.756084698191568]
  Large language model (LLM) distillation aims to transfer the capabilities of a powerful teacher to a smaller student.<n>Direct imitation, however, can also transfer the teacher's systematic bias and errors.<n>We propose Coupled and Learning (CCL), an algorithm that couples teacher calibration with student updates.
  arXiv  Detail & Related papers  (2026-09-15T17:15:40Z)
• Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall [79.59596652272923]
  We propose Switch Distillation, a simple mid-training objective that distills on tokens where the teacher is confident.<n>Relative to standard NTP, it achieves 1.61-1.71x the reasoning performance and 1.13-1.19x the knowledge and commonsense performance.
  arXiv  Detail & Related papers  (2026-09-01T17:00:30Z)
• Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillation [97.02922768457239]
  Context-Matched Distillation (CMD) is a causal DMD framework that aligns teacher supervision with the information available when each target is generated.<n>Prefix Corruption stabilizes training by perturbing unreliable prefixes produced early in training while preserving this target-context alignment.<n> Experiments demonstrate state-of-the-art aggregate performance among autoregressive methods on both short- and long-video benchmarks.
  arXiv  Detail & Related papers  (2026-08-13T15:51:31Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• Skill-Conditioned Gated Self-Distillation for LLM Reasoning [22.02868419200851]
  On-policy self-distillation (SD) improves LLM reasoning by using teacher-side privileged information (PI) to turn sparse verifier outcomes into dense token-level supervision.<n>We propose Skill-Conditioned Gated Self-Distillation (SGSD), which formulates skill-based SD as teacher hypothesis validation rather than unconditional imitation.
  arXiv  Detail & Related papers  (2026-05-27T17:49:52Z)
• Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation [43.020181623750865]
  We propose Monitoring Trajectories and Backtracking when it strays (MOTAB)<n>MOTAB monitors the student's on-policy generation against an adaptive safety boundary.<n>Experiments on the LIMO-v2 and AceReason datasets demonstrate that MOTAB effectively alleviates the dual exposure biases.
  arXiv  Detail & Related papers  (2026-05-19T06:42:52Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)
• Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning [66.52232008796294]
  Prune-OPD dynamically aligns training budgets with supervision quality.<n>It reduces training time by 37.6%--68.0% while preserving, and often improving, performance on challenging benchmarks.
  arXiv  Detail & Related papers  (2026-05-08T14:38:53Z)
• Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization [18.027254451537342]
  Existing self-distillation methods largely reduce learning to KL matching toward the context-augmented teacher model.<n>We propose textbfPreference-textbfBased textbfSelf-textbfDistillation (textbfPBSD), which revisits on-policy self-distillation through a reward-regularized perspective.
  arXiv  Detail & Related papers  (2026-05-06T15:31:50Z)
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents [55.27396165691312]
  We identify a key limitation of vanilla OPD in multi-turn agent settings, which we term Trajectory-Level KL Instability.<n>We propose TCOD, a framework that controls the trajectory depth exposed to the student and progressively expands it with a curriculum schedule.<n> Experimental results across four student-teacher pairs show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD.
  arXiv  Detail & Related papers  (2026-04-27T03:38:27Z)
• Distillation Traps and Guards: A Calibration Knob for LLM Distillability [54.90137955363471]
  We propose a post-hoc calibration method that enables control over a teacher's distillability via reinforcement fine-tuning.<n>Our objective combines task utility, KL anchor, and across-tokenizer calibration reward.<n> Experiments show that students distilled from distillable teachers outperform SFT and KD baselines.
  arXiv  Detail & Related papers  (2026-04-21T01:22:35Z)
• Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models [26.06143154557816]
  Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance.<n>We propose Beta-weighted Knowledge Distillation (Beta-KD), an uncertainty-aware distillation framework that modulates how much the student relies on teacher guidance.
  arXiv  Detail & Related papers  (2026-03-22T22:33:25Z)
• Reinforcement-aware Knowledge Distillation for LLM Reasoning [63.53679456364683]
  Reinforcement learning (RL) post-training has recently driven gains in long chain-of-thought reasoning large language models (LLMs)<n>Most existing knowledge distillation methods are designed for supervised fine-tuning (SFT), relying on fixed teacher traces or teacher-student Kullback-Leibler (KL) divergence-based regularization.<n>We propose RL-aware distillation (RLAD), which performs selective imitation during RL -- guiding the student toward the teacher only when it improves the current policy update.
  arXiv  Detail & Related papers  (2026-02-26T00:20:39Z)
• Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation [4.517380184758914]
  Token-Selective Dual Knowledge Distillation (TSD-KD) is a framework for student-centric distillation.<n>TSD-KD focuses on distilling important tokens for reasoning and encourages the student to explain reasoning in its own words.
  arXiv  Detail & Related papers  (2026-02-25T09:58:49Z)
• Logit Standardization in Knowledge Distillation [83.31794439964033]
  The assumption of a shared temperature between teacher and student implies a mandatory exact match between their logits in terms of logit range and variance. We propose setting the temperature as the weighted standard deviation of logit and performing a plug-and-play Z-score pre-process of logit standardization. Our pre-process enables student to focus on essential logit relations from teacher rather than requiring a magnitude match, and can improve the performance of existing logit-based distillation methods.
  arXiv  Detail & Related papers  (2024-03-03T07:54:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.