Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations

• URL: http://arxiv.org/abs/2607.11557v2
• Date: Wed, 22 Jul 2026 08:11:20 GMT
• Title: Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations
• Abstract summary: Shift-Augmented Knowledge Distillation (SAKD) is a simple yet effective framework that leverages the student's evolving features as a dynamic condition for perturbation generation.<n>SAKD consistently outperforms random perturbation methods and achieves accuracy on par with two-stage approaches.
• Score: 6.992477865573147
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge distillation (KD) typically relies on the fixed perspective of a single teacher, limiting the diversity of supervisory signals. While multi-teacher distillation addresses this by aggregating knowledge from multiple models, it incurs prohibitive computational and storage costs. To balance efficiency and diversity, recent research has focused on generating virtual views from a single teacher. However, existing methods face a trade-off: random perturbation approaches offer efficiency but lack controlled diversity, while structured augmentation methods require multi-stage training and incur linear parameter growth. We observe that this trade-off stems from a common design choice: using the teacher's strong but static features to generate views. Instead, we propose Shift-Augmented Knowledge Distillation (SAKD), a simple yet effective framework that leverages the student's evolving features as a dynamic condition for perturbation generation. This shift in perspective enables single-stage training while producing adaptive, diverse views through a parameter-free cyclic shift. Extensive experiments on CIFAR-100 and ImageNet demonstrate that SAKD consistently outperforms random perturbation methods and achieves accuracy on par with two-stage approaches, while using significantly fewer parameters and eliminating pre-training requirements.

Related papers

• Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors [6.218677961564673]
  We introduce textitMulti-Teacher Bayesian Knowledge Distillation (MT-BKD), where a distilled student model learns from multiple teachers within the Bayesian framework.<n>An entropy-based weighting mechanism adaptively adjusts each teacher's influence, allowing the student to combine multiple sources of expertise effectively.<n>We validate MT-BKD on both synthetic and real-world tasks, including protein subcellular location prediction and image classification.
  arXiv  Detail & Related papers  (2026-05-27T05:03:24Z)
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• Teaching Sarcasm: Few-Shot Multimodal Sarcasm Detection via Distillation to a Parameter-Efficient Student [2.8074191213147652]
  Multimodal sarcasm detection is challenging in low-resource settings.<n>We propose PEKD, a unified framework that enhances PEFT methods via distillation from an expert model trained on large-scale sarcasm data.
  arXiv  Detail & Related papers  (2025-10-29T09:14:41Z)
• Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity [20.479130509494272]
  Knowledge Distillation (KD) aims to train a lightweight student model by transferring knowledge from a large, high-capacity teacher.<n>Recent studies have shown that leveraging diverse teacher perspectives can significantly improve distillation performance.<n>We propose a novel cost-efficient knowledge augmentation method for KD that generates diverse multi-views by attaching multiple branches to a single teacher.
  arXiv  Detail & Related papers  (2025-10-26T01:41:08Z)
• AdaSwitch: Adaptive Switching Generation for Knowledge Distillation [58.647880811071495]
  Small language models (SLMs) are crucial for applications with strict latency and computational constraints.<n>We propose AdaSwitch, a novel approach that combines on-policy and off-policy generation at the token level.<n>AdaSwitch consistently improves accuracy, offering a practical and effective method for distilling SLMs with acceptable additional overhead.
  arXiv  Detail & Related papers  (2025-10-09T06:38:37Z)
• Intra-class Patch Swap for Self-Distillation [3.282914142012984]
  We propose a teacher-free distillation framework based on a single student network.<n>Our approach is built on a simple yet highly effective augmentation, called intra-class patch swap augmentation.<n>Our method consistently outperforms both existing self-distillation baselines and conventional teacher-based KD approaches.
  arXiv  Detail & Related papers  (2025-05-20T09:30:19Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only.<n> Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• Multi-Granularity Semantic Revision for Large Language Model Distillation [66.03746866578274]
  We propose a multi-granularity semantic revision method for LLM distillation. At the sequence level, we propose a sequence correction and re-generation strategy. At the token level, we design a distribution adaptive clipping Kullback-Leibler loss as the distillation objective function. At the span level, we leverage the span priors of a sequence to compute the probability correlations within spans, and constrain the teacher and student's probability correlations to be consistent.
  arXiv  Detail & Related papers  (2024-07-14T03:51:49Z)
• Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners [102.20090188997301]
  We explore how to obtain a model that combines Contrastive Learning (CL) and Masked Image Modeling (MIM) strengths. In order to better obtain both discrimination and diversity, we propose a simple but effective Hybrid Distillation strategy. Experiment results prove that Hybrid Distill can achieve superior performance on different benchmarks.
  arXiv  Detail & Related papers  (2023-06-28T02:19:35Z)
• Class-Balanced Distillation for Long-Tailed Visual Recognition [100.10293372607222]
  Real-world imagery is often characterized by a significant imbalance of the number of images per class, leading to long-tailed distributions. In this work, we introduce a new framework, by making the key observation that a feature representation learned with instance sampling is far from optimal in a long-tailed setting. Our main contribution is a new training method, that leverages knowledge distillation to enhance feature representations.
  arXiv  Detail & Related papers  (2021-04-12T08:21:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.