Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Model

• URL: http://arxiv.org/abs/2602.02107v1
• Date: Mon, 02 Feb 2026 13:52:15 GMT
• Title: Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Model
• Authors: Yu Wang, Chuanguang Yang, Zhulin An, Weilun Feng, Jiarui Zhao, Chengqing Yu, Libo Huang, Boyu Diao, Yongjun Xu,
• Abstract summary: We propose teacher-guided student Diffusion Self-KD, dubbed as DSKD.<n>We leverage the teacher classifier to guide the sampling process of denoising student features through a light-weight diffusion model.<n>The denoised student features teacher knowledge and could be regarded as a teacher role.
• Score: 35.920639111656534
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing Knowledge Distillation (KD) methods often align feature information between teacher and student by exploring meaningful feature processing and loss functions. However, due to the difference in feature distributions between the teacher and student, the student model may learn incompatible information from the teacher. To address this problem, we propose teacher-guided student Diffusion Self-KD, dubbed as DSKD. Instead of the direct teacher-student alignment, we leverage the teacher classifier to guide the sampling process of denoising student features through a light-weight diffusion model. We then propose a novel locality-sensitive hashing (LSH)-guided feature distillation method between the original and denoised student features. The denoised student features encapsulate teacher knowledge and could be regarded as a teacher role. In this way, our DSKD method could eliminate discrepancies in mapping manners and feature distributions between the teacher and student, while learning meaningful knowledge from the teacher. Experiments on visual recognition tasks demonstrate that DSKD significantly outperforms existing KD methods across various models and datasets. Our code is attached in supplementary material.

Related papers

• Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling [81.00825302340984]
  We introduce Speculative Knowledge Distillation (SKD) to generate high-quality training data on-the-fly.<n>In SKD, the student proposes tokens, and the teacher replaces poorly ranked ones based on its own distribution.<n>We evaluate SKD on various text generation tasks, including translation, summarization, math, and instruction following.
  arXiv  Detail & Related papers  (2024-10-15T06:51:25Z)
• Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition [58.41784639847413]
  Human emotion is a complex phenomenon conveyed and perceived through facial expressions, vocal tones, body language, and physiological signals. In this paper, a multi-teacher PKD (MT-PKDOT) method with self-distillation is introduced to align diverse teacher representations before distilling them to the student. Results indicate that our proposed method can outperform SOTA PKD methods.
  arXiv  Detail & Related papers  (2024-08-16T22:11:01Z)
• Improving Knowledge Distillation with Teacher's Explanation [14.935696904019146]
  We introduce a novel Knowledge Explaining Distillation (KED) framework. KED allows the student to learn not only from the teacher's predictions but also from the teacher's explanations. Our experiments over a variety of datasets show that KED students can substantially outperform KD students of similar complexity.
  arXiv  Detail & Related papers  (2023-10-04T04:18:01Z)
• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes [44.97759066341107]
  Generalized Knowledge Distillation (GKD) trains the student on its self-generated output sequences by leveraging feedback from the teacher. We demonstrate the efficacy of GKD for distilling auto-regressive language models on summarization, translation, and arithmetic reasoning tasks.
  arXiv  Detail & Related papers  (2023-06-23T17:56:26Z)
• Knowledge Diffusion for Distillation [53.908314960324915]
  The representation gap between teacher and student is an emerging topic in knowledge distillation (KD) We state that the essence of these methods is to discard the noisy information and distill the valuable information in the feature. We propose a novel KD method dubbed DiffKD, to explicitly denoise and match features using diffusion models.
  arXiv  Detail & Related papers  (2023-05-25T04:49:34Z)
• Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation [70.92135839545314]
  We propose the dynamic prior knowledge (DPK), which integrates part of teacher's features as the prior knowledge before the feature distillation. Our DPK makes the performance of the student model positively correlated with that of the teacher model, which means that we can further boost the accuracy of students by applying larger teachers.
  arXiv  Detail & Related papers  (2022-06-13T11:52:13Z)
• Learning to Teach with Student Feedback [67.41261090761834]
  Interactive Knowledge Distillation (IKD) allows the teacher to learn to teach from the feedback of the student. IKD trains the teacher model to generate specific soft target at each training step for a certain student. Joint optimization for both teacher and student is achieved by two iterative steps.
  arXiv  Detail & Related papers  (2021-09-10T03:01:01Z)
• Fixing the Teacher-Student Knowledge Discrepancy in Distillation [72.4354883997316]
  We propose a novel student-dependent distillation method, knowledge consistent distillation, which makes teacher's knowledge more consistent with the student. Our method is very flexible that can be easily combined with other state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-03-31T06:52:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.