Rethinking Reverse KL as Adaptive Entropy Distillation

• URL: http://arxiv.org/abs/2608.14685v1
• Date: Wed, 05 Aug 2026 04:27:37 GMT
• Title: Rethinking Reverse KL as Adaptive Entropy Distillation
• Abstract summary: We propose textbfAdaptive Entropy Distillation (AED), which uses the teacher's entropy to dynamically calibrate token-level imitation strength.<n>AED achieves superior overall performance and generally improves teacher--student distributional and entropy alignment.
• Score: 18.957879192719762
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge distillation (KD) is widely used to transfer the capabilities of large language models (LLMs) to smaller students, but existing objectives often struggle to balance faithful imitation and robust generation. In particular, existing methods mainly combine FKL and RKL, overlooking that RKL itself provides a mechanism for adjusting the student's imitation strength. Motivated by this, we revisit on-policy Reverse Kullback-Leibler (RKL) distillation and decompose its objective into a teacher-fitting term and a student-entropy term, without introducing an explicit FKL branch. We show theoretically that the token-level optimal student distribution corresponds to a tempered variant of the teacher distribution, where the adaptive weight controls the trade-off between mode-seeking and uncertainty preservation. Guided by this insight, we propose \textbf{Adaptive Entropy Distillation (AED)}, which uses the teacher's entropy to dynamically calibrate token-level imitation strength. Experiments on instruction-following and mathematical reasoning benchmarks demonstrate that AED achieves superior overall performance and generally improves teacher--student distributional and entropy alignment.

Related papers

• Purified OPSD: On-Policy Self-Distillation Without Losing How to Think [52.879387744920415]
  We find that on-policy self-distillation (OPSD) consistently fails on long chain-of-thought (long-CoT) reasoning models.<n>We propose a two-step solution: first, we construct a reference-only teacher to isolate the non-transferable component of the supervision signal.<n>Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution.
  arXiv  Detail & Related papers  (2026-07-02T14:33:07Z)
• Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning [43.9367673156851]
  On-policy self-distillation trains a reasoning model on its own rollouts while a teacher provides dense token-level supervision.<n>We propose EGRSD (Entropy-Guided Reinforced Self-Distillation), which unifies token-level updates through three signals.<n> CL-EGRSD, a causal-lookratio variant that distinguishes sustained high-entropy spans from transient high-entropy positions.
  arXiv  Detail & Related papers  (2026-05-13T09:38:20Z)
• Teacher-Guided Policy Optimization for LLM Distillation [90.49982387646861]
  Teacher-Guided Policy Optimization (TGPO) is an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout.<n>Experiments on complex reasoning benchmarks demonstrate that TGPO significantly outperforms standard baselines and is robust to different teachers.
  arXiv  Detail & Related papers  (2026-05-13T09:20:03Z)
• Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization [18.027254451537342]
  Existing self-distillation methods largely reduce learning to KL matching toward the context-augmented teacher model.<n>We propose textbfPreference-textbfBased textbfSelf-textbfDistillation (textbfPBSD), which revisits on-policy self-distillation through a reward-regularized perspective.
  arXiv  Detail & Related papers  (2026-05-06T15:31:50Z)
• Reinforcement-aware Knowledge Distillation for LLM Reasoning [63.53679456364683]
  Reinforcement learning (RL) post-training has recently driven gains in long chain-of-thought reasoning large language models (LLMs)<n>Most existing knowledge distillation methods are designed for supervised fine-tuning (SFT), relying on fixed teacher traces or teacher-student Kullback-Leibler (KL) divergence-based regularization.<n>We propose RL-aware distillation (RLAD), which performs selective imitation during RL -- guiding the student toward the teacher only when it improves the current policy update.
  arXiv  Detail & Related papers  (2026-02-26T00:20:39Z)
• Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation [50.19746127327559]
  We propose a new tail-aware divergence that decouples the contribution of the teacher model's top-K predicted probabilities from that of lower-probability predictions.<n> Experimental results demonstrate that our modified distillation method yields competitive performance in both pre-training and supervised distillation of decoder models.
  arXiv  Detail & Related papers  (2026-02-24T11:54:06Z)
• Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation [57.524909883706556]
  On-policy distillation (OPD) has demonstrated strong empirical gains in improving student performance.<n>This work introduces a flexible reference model and a reward scaling factor that controls the relative weight of the reward term against the KL regularization.<n>In particular, in the setting where we merge the knowledge from different domain experts, obtained by applying domain-specific RL to the same student model, ExOPD enables the student to even surpass the teacher's performance boundary.
  arXiv  Detail & Related papers  (2026-02-12T16:14:29Z)
• REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency [0.0]
  We introduce REDistill, a principled framework grounded in robust statistics.<n>Redistill replaces the standard KD objective with a power divergence loss, a generalization of KL divergence.<n>Experiments on CIFAR-100 and ImageNet-1k demonstrate that REDistill consistently improves student accuracy in diverse teacher-student architectures.
  arXiv  Detail & Related papers  (2026-02-04T15:50:53Z)
• ToDi: Token-wise Distillation via Fine-Grained Divergence Control [9.958797874295355]
  Token-wise Distillation (ToDi) is a novel method that adaptively combines Forward KL and Reverse KL per token using a sigmoid-based weighting function.<n>ToDi consistently outperforms recent distillation baselines using uniform or less granular strategies.
  arXiv  Detail & Related papers  (2025-05-22T06:51:16Z)
• Multi-Granularity Semantic Revision for Large Language Model Distillation [66.03746866578274]
  We propose a multi-granularity semantic revision method for LLM distillation. At the sequence level, we propose a sequence correction and re-generation strategy. At the token level, we design a distribution adaptive clipping Kullback-Leibler loss as the distillation objective function. At the span level, we leverage the span priors of a sequence to compute the probability correlations within spans, and constrain the teacher and student's probability correlations to be consistent.
  arXiv  Detail & Related papers  (2024-07-14T03:51:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.