Efficient Knowledge Distillation via Curriculum Extraction

• URL: http://arxiv.org/abs/2503.17494v1
• Date: Fri, 21 Mar 2025 19:09:41 GMT
• Title: Efficient Knowledge Distillation via Curriculum Extraction
• Authors: Shivam Gupta, Sushrut Karmalkar,
• Abstract summary: We show that a curriculum can be emphextracted from just the fully trained teacher network, and that this extracted curriculum can give similar efficiency benefits to those of progressive distillation.<n>Our scheme significantly outperforms one-shot distillation and achieves a performance similar to that of progressive distillation for learning sparse parities with two-layer networks.
• Score: 9.320038077848709
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge distillation is a technique used to train a small student network using the output generated by a large teacher network, and has many empirical advantages~\citep{Hinton2015DistillingTK}. While the standard one-shot approach to distillation only uses the output of the final teacher network, recent work~\citep{panigrahi2024progressive} has shown that using intermediate checkpoints from the teacher's training process as an implicit ``curriculum'' for progressive distillation can significantly speed up training. However, such schemes require storing these checkpoints, and often require careful selection of the intermediate checkpoints to train on, which can be impractical for large-scale training. In this paper, we show that a curriculum can be \emph{extracted} from just the fully trained teacher network, and that this extracted curriculum can give similar efficiency benefits to those of progressive distillation. Our extraction scheme is natural; we use a random projection of the hidden representations of the teacher network to progressively train the student network, before training using the output of the full network. We show that our scheme significantly outperforms one-shot distillation and achieves a performance similar to that of progressive distillation for learning sparse parities with two-layer networks, and provide theoretical guarantees for this setting. Additionally, we show that our method outperforms one-shot distillation even when using transformer-based architectures, both for sparse-parity learning, and language modeling tasks.

Related papers

• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only. Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• Towards Training One-Step Diffusion Models Without Distillation [72.80423908458772]
  We show that one-step generative models can be trained directly without this distillation process.<n>We propose a family of distillation methods that achieve competitive results without relying on score estimation.
  arXiv  Detail & Related papers  (2025-02-11T23:02:14Z)
• Contrastive Representation Distillation via Multi-Scale Feature Decoupling [0.49157446832511503]
  Knowledge distillation is a technique aimed at enhancing the performance of a smaller student network without increasing its parameter size. We introduce multi-scale decoupling in the feature transfer process for the first time, where the decoupled local features are individually processed and integrated with contrastive learning. Our approach not only reduces computational costs but also enhances efficiency, enabling performance improvements for the student network using only single-batch samples.
  arXiv  Detail & Related papers  (2025-02-09T10:03:18Z)
• Progressive distillation induces an implicit curriculum [44.528775476168654]
  A better teacher does not always yield a better student, to which a common mitigation is to use additional supervision from several teachers. One empirically validated variant of this principle is progressive distillation, where the student learns from successive intermediate checkpoints of the teacher. Using sparse parity as a sandbox, we identify an implicit curriculum as one mechanism through which progressive distillation accelerates the student's learning.
  arXiv  Detail & Related papers  (2024-10-07T19:49:24Z)
• Knowledge Distillation Meets Open-Set Semi-Supervised Learning [69.21139647218456]
  We propose a novel em modelname (bfem shortname) method dedicated for distilling representational knowledge semantically from a pretrained teacher to a target student. At the problem level, this establishes an interesting connection between knowledge distillation with open-set semi-supervised learning (SSL) Our shortname outperforms significantly previous state-of-the-art knowledge distillation methods on both coarse object classification and fine face recognition tasks.
  arXiv  Detail & Related papers  (2022-05-13T15:15:27Z)
• Representation Consolidation for Training Expert Students [54.90754502493968]
  We show that a multi-head, multi-task distillation method is sufficient to consolidate representations from task-specific teacher(s) and improve downstream performance. Our method can also combine the representational knowledge of multiple teachers trained on one or multiple domains into a single model.
  arXiv  Detail & Related papers  (2021-07-16T17:58:18Z)
• Students are the Best Teacher: Exit-Ensemble Distillation with Multi-Exits [25.140055086630838]
  This paper proposes a novel knowledge distillation-based learning method to improve the classification performance of convolutional neural networks (CNNs) Unlike the conventional notion of distillation where teachers only teach students, we show that students can also help other students and even the teacher to learn better.
  arXiv  Detail & Related papers  (2021-04-01T07:10:36Z)
• Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation [12.097302014936655]
  This paper proposes a novel self-knowledge distillation method, Feature Refinement via Self-Knowledge Distillation (FRSKD) Our proposed method, FRSKD, can utilize both soft label and feature-map distillations for the self-knowledge distillation. We demonstrate the effectiveness of FRSKD by enumerating its performance improvements in diverse tasks and benchmark datasets.
  arXiv  Detail & Related papers  (2021-03-15T10:59:43Z)
• Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition [79.60708268515293]
  This paper explores how to train small and efficient networks for action recognition. We propose two distillation strategies in the frequency domain, namely the feature spectrum and parameter distribution distillations respectively. Our method can achieve higher performance than state-of-the-art methods with the same backbone.
  arXiv  Detail & Related papers  (2020-09-15T07:29:57Z)
• Self-supervised Knowledge Distillation for Few-shot Learning [123.10294801296926]
  Few shot learning is a promising learning paradigm due to its ability to learn out of order distributions quickly with only a few samples. We propose a simple approach to improve the representation capacity of deep neural networks for few-shot learning tasks. Our experiments show that, even in the first stage, self-supervision can outperform current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-17T11:27:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.