Rethinking the Role of Temperature in Large Language Model Distillation

• URL: http://arxiv.org/abs/2606.00306v1
• Date: Fri, 29 May 2026 19:32:21 GMT
• Title: Rethinking the Role of Temperature in Large Language Model Distillation
• Abstract summary: Reverse Kullback-Leibler (RKL) divergence is widely favored over forward KL (FKL) in large language models (LLM) distillation.<n>In this work, we show that it fundamentally changes the comparison between FKL and RKL.<n>Our analysis reveals an asymmetric effect: temperature substantially enriches FKL with non-dominant token signals, whereas it mainly rescales RKL gradients.
• Score: 5.957171492626586
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reverse Kullback-Leibler (RKL) divergence is widely favored over forward KL (FKL) in large language models (LLM) distillation, yet this preference is largely based on comparisons that omit the temperature $τ$, overlooking its central role in softening teacher distributions and improving knowledge transfer. In this work, we revisit temperature in LLM distillation and show that it fundamentally changes the comparison between FKL and RKL. Our analysis reveals an asymmetric effect: temperature substantially enriches FKL with non-dominant token signals, whereas it mainly rescales RKL gradients, causing FKL to benefit much more from $τ$ scaling than RKL. This asymmetry overturns the standard empirical conclusion: although RKL outperforms FKL at $τ=1$, FKL consistently surpasses RKL at higher temperatures across instruction-following benchmarks. Moreover, the impact of temperature is not limited to FKL; it improves a broader family of distillation objectives, enabling simple KL-based methods to achieve competitive performance against recent state-of-the-art LLM distillation approaches.

Related papers

• ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation [5.7913606195390885]
  Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs)<n>We analyze the complementary roles of forward and reverse KL divergence (FKL/RKL) in distribution alignment from theoretical and empirical perspectives.<n>We propose a reinforcement-learning-based adaptive KL-weighted distillation framework, in which a policy network dynamically assigns weights to FKL and RKL based on teacher-student distributional characteristics.
  arXiv  Detail & Related papers  (2026-06-29T07:05:56Z)
• Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation [15.564152482735652]
  We show that the prevailing paradigms, off-policy distillation and on-policy distillation, implicitly couple two choices: prefix source and token-level KL direction.<n>We propose KL mixing and an entropy-gated length curriculum.
  arXiv  Detail & Related papers  (2026-05-16T06:05:27Z)
• Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation [6.256286464835529]
  We propose Diversity-aware RKL (DRKL), which removes this gradient effect and strengthens non-target supervision while preserving the optimization benefits of RKL.<n>DRKL consistently outperforms FKL, RKL, and other state-of-the-art distillation objectives, achieving better performance and a superior fidelity-diversity trade-off.
  arXiv  Detail & Related papers  (2026-03-31T20:39:47Z)
• Reinforcement-aware Knowledge Distillation for LLM Reasoning [63.53679456364683]
  Reinforcement learning (RL) post-training has recently driven gains in long chain-of-thought reasoning large language models (LLMs)<n>Most existing knowledge distillation methods are designed for supervised fine-tuning (SFT), relying on fixed teacher traces or teacher-student Kullback-Leibler (KL) divergence-based regularization.<n>We propose RL-aware distillation (RLAD), which performs selective imitation during RL -- guiding the student toward the teacher only when it improves the current policy update.
  arXiv  Detail & Related papers  (2026-02-26T00:20:39Z)
• A Comedy of Estimators: On KL Regularization in RL Training of LLMs [81.7906270099878]
  reinforcement learning (RL) can substantially improve the reasoning performance of large language models (LLMs)<n>The RL objective for LLM training involves a regularization term, which is the reverse Kullback-Leibler (KL) divergence between the trained policy and the reference policy.<n>Recent works show that prevailing practices for incorporating KL regularization do not provide correct gradients for stated objectives, creating a discrepancy between the objective and its implementation.<n>We study the gradients of several estimators configurations, revealing how design choices shape gradient bias.
  arXiv  Detail & Related papers  (2025-12-26T04:20:58Z)
• A Theoretical Lens for RL-Tuned Language Models via Energy-Based Models [9.324642081509756]
  Large language models (LLMs) trained via KL-regularized reinforcement learning demonstrate strong instruction following, self-correction, and reasoning abilities.<n>We exploit the closed-form energy-based model (EBM) structure of the optimal KL-regularized policy to provide a unified variational analysis of LLMs.
  arXiv  Detail & Related papers  (2025-12-21T13:28:58Z)
• Better Estimation of the Kullback--Leibler Divergence Between Language Models [62.09293217844222]
  Estimating the Kullback--Leibler (KL) divergence between language models has many applications.<n>We introduce a Rao--Blackwellized estimator that is unbiased and provably has variance less than or equal to that of the standard Monte Carlo estimator.
  arXiv  Detail & Related papers  (2025-04-14T18:40:02Z)
• Generalized Kullback-Leibler Divergence Loss [105.66549870868971]
  We prove that the Kullback-Leibler (KL) Divergence loss is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss.<n>Thanks to the decoupled structure of DKL loss, we have identified two areas for improvement.
  arXiv  Detail & Related papers  (2025-03-11T04:43:33Z)
• KL Penalty Control via Perturbation for Direct Preference Optimization [55.24971559149686]
  We propose $varepsilon$-Direct Preference Optimization ($varepsilon$-DPO), which allows adaptive control of the KL penalty strength $beta$ for each preference pair.<n> Experimental results show that the simple criterion of $varepsilon$-DPO for KL penalty relaxation significantly improves DPO compared to most existing direct alignment algorithms.
  arXiv  Detail & Related papers  (2025-02-18T06:44:10Z)
• Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models [18.870276152694245]
  Kullback-Leiber divergence has been widely used in Knowledge Distillation (KD) to compress Large Language Models (LLMs)<n>Contrary to prior assertions, reverse Kullback-Leibler (RKL) divergence is mode-seeking and thus preferable over the mean-seeking forward Kullback-Leibler (FKL) divergence.<n>We propose a simple yet effective Adaptive Kullback-Leiber (AKL) divergence method, which adaptively allocates weights to combine FKL and RKL.
  arXiv  Detail & Related papers  (2024-04-03T11:40:17Z)
• Decoupled Kullback-Leibler Divergence Loss [90.54331083430597]
  We prove that the Kullback-Leibler (KL) Divergence loss is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss. We introduce class-wise global information into KL/DKL to bias from individual samples. The proposed approach achieves new state-of-the-art adversarial robustness on the public leaderboard.
  arXiv  Detail & Related papers  (2023-05-23T11:17:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.