Knowledge Distillation Based on Transformed Teacher Matching

• URL: http://arxiv.org/abs/2402.11148v2
• Date: Thu, 7 Mar 2024 22:41:33 GMT
• Title: Knowledge Distillation Based on Transformed Teacher Matching
• Abstract summary: We study the resulting variant of knowledge distillation (KD) dubbed transformed teacher matching (TTM) By reinterpreting temperature scaling as a power transform of probability distribution, we show that TTM has an inherent R'enyi entropy term in its objective function. Experiment results demonstrate that TTM leads to trained students with better generalization than the original KD.
• Score: 4.567223151852012
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As a technique to bridge logit matching and probability distribution matching, temperature scaling plays a pivotal role in knowledge distillation (KD). Conventionally, temperature scaling is applied to both teacher's logits and student's logits in KD. Motivated by some recent works, in this paper, we drop instead temperature scaling on the student side, and systematically study the resulting variant of KD, dubbed transformed teacher matching (TTM). By reinterpreting temperature scaling as a power transform of probability distribution, we show that in comparison with the original KD, TTM has an inherent R\'enyi entropy term in its objective function, which serves as an extra regularization term. Extensive experiment results demonstrate that thanks to this inherent regularization, TTM leads to trained students with better generalization than the original KD. To further enhance student's capability to match teacher's power transformed probability distribution, we introduce a sample-adaptive weighting coefficient into TTM, yielding a novel distillation approach dubbed weighted TTM (WTTM). It is shown, by comprehensive experiments, that although WTTM is simple, it is effective, improves upon TTM, and achieves state-of-the-art accuracy performance. Our source code is available at https://github.com/zkxufo/TTM.

Related papers

• Temperature-Adaptive Transformed Teacher Matching [0.42970700836450487]
  Transformed Teacher Matching (TTM) clarifies the role of temperature scaling by applying it only to the teacher distribution.<n>We introduce a sample-wise inverse-temperature update for TTM by locally minimizing the Kullback-Leibler divergence between the temperature-scaled teacher distribution and the student's prediction.
  arXiv  Detail & Related papers  (2026-08-29T07:08:26Z)
• Simple-OPD: Demystifying Warm-up for On-policy Distillation [51.693357695326846]
  On-policy distillation (OPD) trains a student on its own rollouts with token-level supervision from teacher models.<n>We demystify warm-up for OPD from both data and training perspectives.
  arXiv  Detail & Related papers  (2026-08-07T04:47:38Z)
• Rethinking Reverse KL as Adaptive Entropy Distillation [18.957879192719762]
  We propose textbfAdaptive Entropy Distillation (AED), which uses the teacher's entropy to dynamically calibrate token-level imitation strength.<n>AED achieves superior overall performance and generally improves teacher--student distributional and entropy alignment.
  arXiv  Detail & Related papers  (2026-08-05T04:27:37Z)
• Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation [50.19746127327559]
  We propose a new tail-aware divergence that decouples the contribution of the teacher model's top-K predicted probabilities from that of lower-probability predictions.<n> Experimental results demonstrate that our modified distillation method yields competitive performance in both pre-training and supervised distillation of decoder models.
  arXiv  Detail & Related papers  (2026-02-24T11:54:06Z)
• PLD: A Choice-Theoretic List-Wise Knowledge Distillation [14.801268227422517]
  We recast knowledge distillation under the Plackett-Luce model by interpreting teacher logits as "worth" scores.<n>We introduce Plackett-Luce Distillation (PLD), a weighted list-wise ranking loss in which the teacher model transfers knowledge of its full ranking of classes.
  arXiv  Detail & Related papers  (2025-06-14T15:31:54Z)
• Test-Time Training Provably Improves Transformers as In-context Learners [49.09821664572445]
  We investigate a gradient-based TTT algorithm for in-context learning. We train a transformer model on the in-context demonstrations provided in the test prompt. As our empirical contribution, we study the benefits of TTT for TabPFN.
  arXiv  Detail & Related papers  (2025-03-14T20:06:37Z)
• Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation [84.38105530043741]
  We propose Warmup-Distill, which aligns the distillation of the student to that of the teacher in advance of distillation. Experiments on the seven benchmarks demonstrate that Warmup-Distill could provide a warmup student more suitable for distillation.
  arXiv  Detail & Related papers  (2025-02-17T12:58:12Z)
• T-SCEND: Test-time Scalable MCTS-enhanced Diffusion Model [7.250494262573953]
  Test-time Scalable MCTS-enhanced Diffusion Model (T-SCEND) is a novel framework that significantly improves diffusion model's reasoning capabilities. T-SCEND integrates the denoising process with a novel hybrid Monte Carlo Tree Search. We demonstrate the effectiveness of T-SCEND's training objective and scalable inference method.
  arXiv  Detail & Related papers  (2025-02-04T04:07:48Z)
• Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition [58.41784639847413]
  Human emotion is a complex phenomenon conveyed and perceived through facial expressions, vocal tones, body language, and physiological signals. In this paper, a multi-teacher PKD (MT-PKDOT) method with self-distillation is introduced to align diverse teacher representations before distilling them to the student. Results indicate that our proposed method can outperform SOTA PKD methods.
  arXiv  Detail & Related papers  (2024-08-16T22:11:01Z)
• Multi-Granularity Semantic Revision for Large Language Model Distillation [66.03746866578274]
  We propose a multi-granularity semantic revision method for LLM distillation. At the sequence level, we propose a sequence correction and re-generation strategy. At the token level, we design a distribution adaptive clipping Kullback-Leibler loss as the distillation objective function. At the span level, we leverage the span priors of a sequence to compute the probability correlations within spans, and constrain the teacher and student's probability correlations to be consistent.
  arXiv  Detail & Related papers  (2024-07-14T03:51:49Z)
• Dynamic Temperature Knowledge Distillation [9.6046915661065]
  Temperature plays a pivotal role in moderating label softness in the realm of knowledge distillation (KD) Traditional approaches often employ a static temperature throughout the KD process. We propose Dynamic Temperature Knowledge Distillation (DTKD) which introduces a dynamic, cooperative temperature control for both teacher and student models simultaneously.
  arXiv  Detail & Related papers  (2024-04-19T08:40:52Z)
• Logit Standardization in Knowledge Distillation [83.31794439964033]
  The assumption of a shared temperature between teacher and student implies a mandatory exact match between their logits in terms of logit range and variance. We propose setting the temperature as the weighted standard deviation of logit and performing a plug-and-play Z-score pre-process of logit standardization. Our pre-process enables student to focus on essential logit relations from teacher rather than requiring a magnitude match, and can improve the performance of existing logit-based distillation methods.
  arXiv  Detail & Related papers  (2024-03-03T07:54:03Z)
• Cosine Similarity Knowledge Distillation for Individual Class Information Transfer [11.544799404018473]
  We introduce a novel Knowledge Distillation (KD) method capable of achieving results on par with or superior to the teacher models performance. We use cosine similarity, a technique in Natural Language Processing (NLP) for measuring the resemblance between text embeddings. We propose a method called cosine similarity weighted temperature (CSWT) to improve the performance.
  arXiv  Detail & Related papers  (2023-11-24T06:34:47Z)
• Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning [73.75282761503581]
  We propose DiffTPT, which leverages pre-trained diffusion models to generate diverse and informative new data. Our experiments on test datasets with distribution shifts and unseen categories demonstrate that DiffTPT improves the zero-shot accuracy by an average of 5.13%.
  arXiv  Detail & Related papers  (2023-08-11T09:36:31Z)
• Debiased Distillation by Transplanting the Last Layer [14.861216810146114]
  Deep models are susceptible to learning spurious correlations, even during the post-processing. We propose a simple knowledge distillation algorithm, coined DeTT (Debiasing by Teacher Transplanting) DeTT successfully debiases the student model, consistently outperforming the baselines in terms of the worst-group accuracy.
  arXiv  Detail & Related papers  (2023-02-22T07:41:09Z)
• DLTTA: Dynamic Learning Rate for Test-time Adaptation on Cross-domain Medical Images [56.72015587067494]
  We propose a novel dynamic learning rate adjustment method for test-time adaptation, called DLTTA. Our method achieves effective and fast test-time adaptation with consistent performance improvement over current state-of-the-art test-time adaptation methods.
  arXiv  Detail & Related papers  (2022-05-27T02:34:32Z)
• ALM-KD: Knowledge Distillation with noisy labels via adaptive loss mixing [25.49637460661711]
  Knowledge distillation is a technique where the outputs of a pretrained model are used for training a student model in a supervised setting. We tackle this problem via the use of an adaptive loss mixing scheme during KD. We demonstrate performance gains obtained using our approach in the standard KD setting as well as in multi-teacher and self-distillation settings.
  arXiv  Detail & Related papers  (2022-02-07T14:53:22Z)
• Knowledge distillation via adaptive instance normalization [52.91164959767517]
  We propose a new knowledge distillation method based on transferring feature statistics from the teacher to the student. Our method goes beyond the standard way of enforcing the mean and variance of the student to be similar to those of the teacher. We show that our distillation method outperforms other state-of-the-art distillation methods over a large set of experimental settings.
  arXiv  Detail & Related papers  (2020-03-09T17:50:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.