Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it

• URL: http://arxiv.org/abs/2608.23752v1
• Date: Mon, 24 Aug 2026 18:40:20 GMT
• Title: Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it
• Abstract summary: Knowledge Distillation (KD) transfers knowledge from a large network (teacher) to a small one (student)<n>This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher.<n>We show that on classic datasets, distilling only the last block is sufficient -- and often best--- whereas fine-grained, data-scarce settings benefit substantially from intermediate supervision.
• Score: 2.915890028770192
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The growing size of Convolutional Neural Networks has led to increasingly large and costly models. Knowledge Distillation (KD) addresses this by transferring knowledge from a large network (teacher) to a small one (student), also reducing the training data required. KD is traditionally applied only at the network's final output. However, its behaviour when applied at intermediate network layers has received little attention. This raises the question of whether intermediate block-wise KD, which provides supervision throughout the network, could offer an advantage under specific conditions, such as few instances per class, which is common in fine-grained datasets. This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks. Across eleven datasets, we show that on classic datasets, distilling only the last block is sufficient -- and often best--, whereas fine-grained, data-scarce settings benefit substantially from intermediate supervision, with even a single additional distillation point narrowing the gap considerably. We further study how this supervision should be guided, exploring configurations of varying granularity and informed by an explainability analysis based on attention maps, Centered Kernel Alignment, and Grad-CAM, alongside the impact of teacher and student fine-tuning strategies. This work shows that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.

Related papers

• WeCKD: Weakly-supervised Chained Distillation Network for Efficient Multimodal Medical Imaging [1.9316515057518757]
  We present a Weakly-supervised Chain-based KD network that redefines knowledge transfer through a structured sequence of interconnected models.<n>Each model in the chain is trained on only a fraction of the dataset and shows that effective learning can be achieved with minimal supervision.<n>The proposed distillation chain resulted in cumulative accuracy gains of up to +23% over a single backbone trained on the same limited data.
  arXiv  Detail & Related papers  (2025-10-16T13:22:51Z)
• Preserving Angles Improves Feature Distillation of Foundation Models [8.572967695281054]
  Preserving similarities between a compress space network and a student image model is presented.<n>It is shown that variety of CossNet datasets, produces accurate with greater robustness on detection benchmarks.<n>This provides a competitive pathway for training on general detection benchmarks.
  arXiv  Detail & Related papers  (2024-11-22T01:48:44Z)
• Attention-guided Feature Distillation for Semantic Segmentation [8.344263189293578]
  This paper showcases the efficacy of a simple yet powerful method for utilizing refined feature maps to transfer attention.<n>The proposed Attention-guided Feature Distillation (AttnFD) method, employs the Convolutional Block Attention Module (CBAM)<n>It achieves state-of-the-art results in terms of improving the mean Intersection over Union (mIoU) of the student network on the PascalVoc 2012, Cityscapes, COCO, and CamVid datasets.
  arXiv  Detail & Related papers  (2024-03-08T16:57:47Z)
• Improving Knowledge Distillation via Regularizing Feature Norm and Direction [16.98806338782858]
  Knowledge distillation (KD) exploits a large well-trained model (i.e., teacher) to train a small student model on the same dataset for the same task. Treating teacher features as knowledge, prevailing methods of knowledge distillation train student by aligning its features with the teacher's, e.g., by minimizing the KL-divergence between their logits or L2 distance between their intermediate features. While it is natural to believe that better alignment of student features to the teacher better distills teacher knowledge, simply forcing this alignment does not directly contribute to the student's performance, e.g.
  arXiv  Detail & Related papers  (2023-05-26T15:05:19Z)
• BD-KD: Balancing the Divergences for Online Knowledge Distillation [11.874952582465601]
  We introduce BD-KD (Balanced Divergence Knowledge Distillation), a framework for logit-based online KD.<n> BD-KD enhances both accuracy and model calibration simultaneously, eliminating the need for post-hoc recalibration techniques.<n>Our method encourages student-centered training by adjusting the conventional online distillation loss on both the student and teacher losses.
  arXiv  Detail & Related papers  (2022-12-25T22:27:32Z)
• CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation [4.182345120164705]
  This paper proposes a new technique called Curriculum Expert Selection for Knowledge Distillation (CES-KD) CES-KD is built upon the hypothesis that a student network should be guided gradually using stratified teaching curriculum. Specifically, our method is a gradual TA-based KD technique that selects a single teacher per input image based on a curriculum driven by the difficulty in classifying the image.
  arXiv  Detail & Related papers  (2022-09-15T21:02:57Z)
• On-Device Domain Generalization [93.79736882489982]
  Domain generalization is critical to on-device machine learning applications. We find that knowledge distillation is a strong candidate for solving the problem. We propose a simple idea called out-of-distribution knowledge distillation (OKD), which aims to teach the student how the teacher handles (synthetic) out-of-distribution data.
  arXiv  Detail & Related papers  (2022-09-15T17:59:31Z)
• InDistill: Information flow-preserving knowledge distillation for model compression [20.88709060450944]
  We introduce InDistill, a method that serves as a warmup stage for Knowledge Distillation (KD) effectiveness.<n>InDistill focuses on transferring critical information flow paths from a heavyweight teacher to a lightweight student.<n>The proposed method is extensively evaluated using various pairs of teacher-student architectures on CIFAR-10, CIFAR-100, and ImageNet datasets.
  arXiv  Detail & Related papers  (2022-05-20T07:40:09Z)
• Knowledge Distillation Meets Open-Set Semi-Supervised Learning [69.21139647218456]
  We propose a novel em modelname (bfem shortname) method dedicated for distilling representational knowledge semantically from a pretrained teacher to a target student. At the problem level, this establishes an interesting connection between knowledge distillation with open-set semi-supervised learning (SSL) Our shortname outperforms significantly previous state-of-the-art knowledge distillation methods on both coarse object classification and fine face recognition tasks.
  arXiv  Detail & Related papers  (2022-05-13T15:15:27Z)
• Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data [56.29595334715237]
  Knowledge distillation(KD) aims to craft a compact student model that imitates the behavior of a pre-trained teacher in a target domain. We introduce a handy yet surprisingly efficacious approach, dubbed astextitMosaicKD. In MosaicKD, this is achieved through a four-player min-max game, in which a generator, a discriminator, a student network, are collectively trained in an adversarial manner.
  arXiv  Detail & Related papers  (2021-10-27T13:01:10Z)
• Distilling Knowledge via Knowledge Review [69.15050871776552]
  We study the factor of connection path cross levels between teacher and student networks, and reveal its great importance. For the first time in knowledge distillation, cross-stage connection paths are proposed. Our finally designed nested and compact framework requires negligible overhead, and outperforms other methods on a variety of tasks.
  arXiv  Detail & Related papers  (2021-04-19T04:36:24Z)
• Wasserstein Contrastive Representation Distillation [114.24609306495456]
  We propose Wasserstein Contrastive Representation Distillation (WCoRD), which leverages both primal and dual forms of Wasserstein distance for knowledge distillation. The dual form is used for global knowledge transfer, yielding a contrastive learning objective that maximizes the lower bound of mutual information between the teacher and the student networks. Experiments demonstrate that the proposed WCoRD method outperforms state-of-the-art approaches on privileged information distillation, model compression and cross-modal transfer.
  arXiv  Detail & Related papers  (2020-12-15T23:43:28Z)
• Efficient Crowd Counting via Structured Knowledge Transfer [122.30417437707759]
  Crowd counting is an application-oriented task and its inference efficiency is crucial for real-world applications. We propose a novel Structured Knowledge Transfer framework to generate a lightweight but still highly effective student network. Our models obtain at least 6.5$times$ speed-up on an Nvidia 1080 GPU and even achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-03-23T08:05:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.