Improving Diversity in Black-box Few-shot Knowledge Distillation

• URL: http://arxiv.org/abs/2604.25795v1
• Date: Tue, 28 Apr 2026 16:03:07 GMT
• Title: Improving Diversity in Black-box Few-shot Knowledge Distillation
• Abstract summary: We propose a novel training scheme for generative adversarial networks.<n>We adaptively select high-confidence images under the teacher's supervision and introduce them to the adversarial learning on-the-fly.<n>We achieve state-of-the-art results among other few-shot KD methods on seven image datasets.
• Score: 21.030238888828915
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge distillation (KD) is a well-known technique to effectively compress a large network (teacher) to a smaller network (student) with little sacrifice in performance. However, most KD methods require a large training set and internal access to the teacher, which are rarely available due to various restrictions. These challenges have originated a more practical setting known as black-box few-shot KD, where the student is trained with few images and a black-box teacher. Recent approaches typically generate additional synthetic images but lack an active strategy to promote their diversity, a crucial factor for student learning. To address these problems, we propose a novel training scheme for generative adversarial networks, where we adaptively select high-confidence images under the teacher's supervision and introduce them to the adversarial learning on-the-fly. Our approach helps expand and improve the diversity of the distillation set, significantly boosting student accuracy. Through extensive experiments, we achieve state-of-the-art results among other few-shot KD methods on seven image datasets. The code is available at https://github.com/votrinhan88/divbfkd.

Related papers

• Heterogeneous and Adept Snapshot Distillation for 3D Semantic Segmentation [71.94615164565589]
  Multi-modal fusion and multi-model ensembling are prevalent in enhancing the performance of 3D semantic segmentation.<n>We propose to transfer the rich knowledge from the multi-modal model (i.e., point clouds and images) and multiple model experts to the point-cloud-based network through knowledge distillation.<n>The resulting Heterogeneous and Adept Snapshot Knowledge Distillation, dubbed HAS-KD, attains state-of-the-art results on ScanNetV2 and S3DIS datasets.
  arXiv  Detail & Related papers  (2026-06-24T01:31:22Z)
• All You Need in Knowledge Distillation Is a Tailored Coordinate System [20.846344563444656]
  Knowledge Distillation (KD) is essential in transferring dark knowledge from a large teacher to a small student network.<n>Existing KD methods rely on a large teacher trained specifically for the target task, which is both very inflexible and inefficient.<n>We argue that a SSL-pretrained model can effectively act as the teacher and its dark knowledge can be captured by the coordinate system.
  arXiv  Detail & Related papers  (2024-12-12T15:56:20Z)
• Discriminative and Consistent Representation Distillation [6.24302896438145]
  Discriminative and Consistent Distillation (DCD)<n>DCD employs a contrastive loss along with a consistency regularization to minimize the discrepancy between the distributions of teacher and student representations.<n>Our method introduces learnable temperature and bias parameters that adapt during training to balance these complementary objectives.
  arXiv  Detail & Related papers  (2024-07-16T14:53:35Z)
• MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution [6.983043882738687]
  We propose a novel Multi-Teacher Knowledge Distillation (MTKD) framework specifically for image super-resolution. It exploits the advantages of multiple teachers by combining and enhancing the outputs of these teacher models. We fully evaluate the effectiveness of the proposed method by comparing it to five commonly used KD methods for image super-resolution.
  arXiv  Detail & Related papers  (2024-04-15T08:32:41Z)
• Revisiting Knowledge Distillation for Autoregressive Language Models [88.80146574509195]
  We propose a simple yet effective adaptive teaching approach (ATKD) to improve the knowledge distillation (KD) The core of ATKD is to reduce rote learning and make teaching more diverse and flexible. Experiments on 8 LM tasks show that, with the help of ATKD, various baseline KD methods can achieve consistent and significant performance gains.
  arXiv  Detail & Related papers  (2024-02-19T07:01:10Z)
• CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation [4.182345120164705]
  This paper proposes a new technique called Curriculum Expert Selection for Knowledge Distillation (CES-KD) CES-KD is built upon the hypothesis that a student network should be guided gradually using stratified teaching curriculum. Specifically, our method is a gradual TA-based KD technique that selects a single teacher per input image based on a curriculum driven by the difficulty in classifying the image.
  arXiv  Detail & Related papers  (2022-09-15T21:02:57Z)
• Black-box Few-shot Knowledge Distillation [55.27881513982002]
  Knowledge distillation (KD) is an efficient approach to transfer the knowledge from a large "teacher" network to a smaller "student" network. We propose a black-box few-shot KD method to train the student with few unlabeled training samples and a black-box teacher. We conduct extensive experiments to show that our method significantly outperforms recent SOTA few/zero-shot KD methods on image classification tasks.
  arXiv  Detail & Related papers  (2022-07-25T12:16:53Z)
• Undistillable: Making A Nasty Teacher That CANNOT teach students [84.6111281091602]
  This paper introduces and investigates a concept called Nasty Teacher: a specially trained teacher network that yields nearly the same performance as a normal one. We propose a simple yet effective algorithm to build the nasty teacher, called self-undermining knowledge distillation.
  arXiv  Detail & Related papers  (2021-05-16T08:41:30Z)
• Knowledge Distillation Thrives on Data Augmentation [65.58705111863814]
  Knowledge distillation (KD) is a general deep neural network training framework that uses a teacher model to guide a student model. Many works have explored the rationale for its success, however, its interplay with data augmentation (DA) has not been well recognized so far. In this paper, we are motivated by an interesting observation in classification: KD loss can benefit from extended training iterations while the cross-entropy loss does not. We show this disparity arises because of data augmentation: KD loss can tap into the extra information from different input views brought by DA.
  arXiv  Detail & Related papers  (2020-12-05T00:32:04Z)
• Heterogeneous Knowledge Distillation using Information Flow Modeling [82.83891707250926]
  We propose a novel KD method that works by modeling the information flow through the various layers of the teacher model. The proposed method is capable of overcoming the aforementioned limitations by using an appropriate supervision scheme during the different phases of the training process.
  arXiv  Detail & Related papers  (2020-05-02T06:56:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.