Temperature-Adaptive Transformed Teacher Matching

• URL: http://arxiv.org/abs/2608.29099v1
• Date: Sat, 29 Aug 2026 07:08:26 GMT
• Title: Temperature-Adaptive Transformed Teacher Matching
• Abstract summary: Transformed Teacher Matching (TTM) clarifies the role of temperature scaling by applying it only to the teacher distribution.<n>We introduce a sample-wise inverse-temperature update for TTM by locally minimizing the Kullback-Leibler divergence between the temperature-scaled teacher distribution and the student's prediction.
• Score: 0.42970700836450487
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Temperature scaling is a core component of knowledge distillation, yet its role and effect are still not fully understood. Transformed Teacher Matching (TTM) clarifies the role of temperature scaling by applying it only to the teacher distribution and interpreting the resulting objective as standard distillation with an implicit Rényi entropy regularization on the student. However, TTM still relies on a fixed temperature and does not specify how the teacher-side temperature should be adapted for individual samples. In this paper, we introduce a sample-wise inverse-temperature update for TTM by locally minimizing the Kullback-Leibler divergence between the temperature-scaled teacher distribution and the student's prediction. We derive closed-form first and second derivatives with respect to the inverse temperature, and show that they can be expressed using variance and covariance statistics of centered teacher and student logits under the transformed teacher weighting. This yields an efficient curvature-aware update that requires one softmax evaluation and a constant number of class-wise weighted sums. Experiments on standard image classification distillation benchmarks show that our temperature adaptation generally improves TTM and WTTM, while remaining competitive with or outperforming prior temperature-adaptive distillation baselines.

Related papers

• TeMo: Temperature Modulation for Multimodal Contrastive Learning [69.50376336775496]
  Contrastive learning approaches achieve strong performance by training models to bring similar samples closer while pushing dissimilar samples apart.<n>We introduce TeMo, Temperature Modulation framework, a similarity-based modulation approach that adaptively adjusts the temperature for each positive-negative pair according to their similarity.<n>Our approach seamlessly integrates temperature-modulated multimodal and unimodal losses with the standard multimodal contrastive loss by gradually transitioning between them.
  arXiv  Detail & Related papers  (2026-09-07T14:23:02Z)
• Diversify Diffusion with Temperature Sampling and Variance-Corrective Time Shifting [57.75750707482832]
  We introduce variance-corrective time shifting, a training-free fix that queries the network at a shifted timestep and scales the resulting score by $$, canceling the variance inflation while preserving the mode reweighting.<n>We demonstrate consistent gains at minimal cost to sample quality and condition fidelity across DiT, Stable Diffusion and Motion Diffusion models.
  arXiv  Detail & Related papers  (2026-07-12T17:32:29Z)
• Consistently Informative Soft-Label Temperature for Knowledge Distillation [6.328321206328517]
  Knowledge distillation (KD) transfers knowledge from a high-capacity teacher to a compact student by matching their predictive distributions.<n>Standard fixed-temperature design is inherently sample-agnostic.<n>We propose CIST (Consistently Informative Soft-label Temperature), which assigns separate sample-wise adaptive temperatures to the teacher and student.
  arXiv  Detail & Related papers  (2026-05-19T18:11:54Z)
• MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data [64.78447637450937]
  We propose Multi-Modal Temperature and Margin Schedules (MM-TS), extending the concept of uni-modal temperature scheduling to multi-modal contrastive learning.<n>Our method dynamically adjusts the temperature in the contrastive loss during training, modulating the attraction and repulsion forces in the multi-modal setting.
  arXiv  Detail & Related papers  (2026-03-09T10:29:50Z)
• Optimal Attention Temperature Enhances In-Context Learning under Distribution Shift [8.135786025034626]
  This paper provides the first theoretical and empirical study of attention temperature for ICL under distribution shift.<n>We prove that shifts in input covariance or label noise substantially impair ICL, but that an optimal attention temperature exists which minimizes this error.<n>Our results establish attention temperature as a principled and powerful mechanism for improving the robustness of ICL in pretrained Transformers.
  arXiv  Detail & Related papers  (2025-11-03T07:18:27Z)
• Analytical Softmax Temperature Setting from Feature Dimensions for Model- and Domain-Robust Classification [0.0]
  In deep learning-based classification tasks, the temperature parameter $T$ critically influences the output distribution and overall performance.<n>This study presents a novel theoretical insight that the optimal temperature $T*$ is uniquely determined by the dimensionality of the feature representations.<n>We develop an empirical formula to estimate $T*$ without additional training while also introducing a corrective scheme to refine $T*$ based on the number of classes and task complexity.
  arXiv  Detail & Related papers  (2025-04-22T05:14:38Z)
• Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation [84.38105530043741]
  We propose Warmup-Distill, which aligns the distillation of the student to that of the teacher in advance of distillation.<n>Experiments on the seven benchmarks demonstrate that Warmup-Distill could provide a warmup student more suitable for distillation.
  arXiv  Detail & Related papers  (2025-02-17T12:58:12Z)
• Calibrating Language Models with Adaptive Temperature Scaling [58.056023173579625]
  We introduce Adaptive Temperature Scaling (ATS), a post-hoc calibration method that predicts a temperature scaling parameter for each token prediction. ATS improves calibration by over 10-50% across three downstream natural language evaluation benchmarks compared to prior calibration methods.
  arXiv  Detail & Related papers  (2024-09-29T22:54:31Z)
• Logit Standardization in Knowledge Distillation [83.31794439964033]
  The assumption of a shared temperature between teacher and student implies a mandatory exact match between their logits in terms of logit range and variance. We propose setting the temperature as the weighted standard deviation of logit and performing a plug-and-play Z-score pre-process of logit standardization. Our pre-process enables student to focus on essential logit relations from teacher rather than requiring a magnitude match, and can improve the performance of existing logit-based distillation methods.
  arXiv  Detail & Related papers  (2024-03-03T07:54:03Z)
• Knowledge Distillation Based on Transformed Teacher Matching [4.567223151852012]
  We study the resulting variant of knowledge distillation (KD) dubbed transformed teacher matching (TTM) By reinterpreting temperature scaling as a power transform of probability distribution, we show that TTM has an inherent R'enyi entropy term in its objective function. Experiment results demonstrate that TTM leads to trained students with better generalization than the original KD.
  arXiv  Detail & Related papers  (2024-02-17T00:28:06Z)
• Knowledge distillation via adaptive instance normalization [52.91164959767517]
  We propose a new knowledge distillation method based on transferring feature statistics from the teacher to the student. Our method goes beyond the standard way of enforcing the mean and variance of the student to be similar to those of the teacher. We show that our distillation method outperforms other state-of-the-art distillation methods over a large set of experimental settings.
  arXiv  Detail & Related papers  (2020-03-09T17:50:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.