Diverse Image Priors for Black-box Data-free Knowledge Distillation

• URL: http://arxiv.org/abs/2604.25794v1
• Date: Tue, 28 Apr 2026 16:02:27 GMT
• Title: Diverse Image Priors for Black-box Data-free Knowledge Distillation
• Abstract summary: In decentralized or secure AI ecosystems, privacy regulations and proprietary interests often restrict access to the teacher's interface and original datasets.<n>We propose Diverse Image Priors Knowledge Distillation (DIP-KD), a framework that addresses these challenges through a three-phase collaborative pipeline.<n>Our evaluation shows that DIP-KD achieves state-of-the-art performance, with ablations confirming data diversity as critical for knowledge acquisition in restricted AI environments.
• Score: 26.477721699257817
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge distillation (KD) represents a vital mechanism to transfer expertise from complex teacher networks to efficient student models. However, in decentralized or secure AI ecosystems, privacy regulations and proprietary interests often restrict access to the teacher's interface and original datasets. These constraints define a challenging black-box data-free KD scenario where only top-1 predictions and no training data are available. While recent approaches utilize synthetic data, they still face limitations in data diversity and distillation signals. We propose Diverse Image Priors Knowledge Distillation (DIP-KD), a framework that addresses these challenges through a three-phase collaborative pipeline: (1) Synthesis of image priors to capture diverse visual patterns and semantics; (2) Contrast to enhance the collective distinction between synthetic samples via contrastive learning; and (3) Distillation via a novel primer student that enables soft-probability KD. Our evaluation across 12 benchmarks shows that DIP-KD achieves state-of-the-art performance, with ablations confirming data diversity as critical for knowledge acquisition in restricted AI environments.

Related papers

• Bridging Cognitive Gap: Hierarchical Description Learning for Artistic Image Aesthetics Assessment [51.40989269202702]
  aesthetic quality assessment task is crucial for developing a human-aligned quantitative evaluation system for AIGC.<n>We propose ArtQuant, an aesthetics assessment framework for artistic images which couples isolated aesthetic dimensions through description generation.<n>Our approach achieves epoch state-of-the-art performance on several datasets while requiring only 33% of conventional trainings.
  arXiv  Detail & Related papers  (2025-12-29T12:18:26Z)
• Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillation [13.970649308150895]
  Knowledge distillation (KD) methods can transfer knowledge of a parameter-heavy teacher model to a light-weight student model.<n>We propose a feature KD framework for training the student's backbone using feature-based losses exclusively.<n>We demonstrate our KD method achieves state-of-the-art performance, delivering top-1 accuracy boosts of up to 15% over standard approaches.
  arXiv  Detail & Related papers  (2025-11-18T23:50:31Z)
• WeCKD: Weakly-supervised Chained Distillation Network for Efficient Multimodal Medical Imaging [1.9316515057518757]
  We present a Weakly-supervised Chain-based KD network that redefines knowledge transfer through a structured sequence of interconnected models.<n>Each model in the chain is trained on only a fraction of the dataset and shows that effective learning can be achieved with minimal supervision.<n>The proposed distillation chain resulted in cumulative accuracy gains of up to +23% over a single backbone trained on the same limited data.
  arXiv  Detail & Related papers  (2025-10-16T13:22:51Z)
• Conditional Pseudo-Supervised Contrast for Data-Free Knowledge Distillation [7.195870730342018]
  We propose a novel learning paradigm, i.e., conditional pseudo-supervised contrast for data-free knowledge distillation(CPSC-DFKD)<n>The primary innovations of CPSC-DFKD are: (1) introducing a conditional generative adversarial network to synthesize category-specific diverse images for pseudo-supervised learning, (2) improving the modules of the generator to distinguish the distributions of different categories, and (3) proposing pseudo-supervised contrastive learning based on teacher and student views to enhance diversity.
  arXiv  Detail & Related papers  (2025-10-03T13:34:19Z)
• Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition [58.41784639847413]
  Human emotion is a complex phenomenon conveyed and perceived through facial expressions, vocal tones, body language, and physiological signals. In this paper, a multi-teacher PKD (MT-PKDOT) method with self-distillation is introduced to align diverse teacher representations before distilling them to the student. Results indicate that our proposed method can outperform SOTA PKD methods.
  arXiv  Detail & Related papers  (2024-08-16T22:11:01Z)
• Distilling Privileged Multimodal Information for Expression Recognition using Optimal Transport [46.91791643660991]
  Deep learning models for multimodal expression recognition have reached remarkable performance in controlled laboratory environments. These models struggle in the wild because of the unavailability and quality of modalities used for training. In practice, only a subset of the training-time modalities may be available at test time. Learning with privileged information enables models to exploit data from additional modalities that are only available during training.
  arXiv  Detail & Related papers  (2024-01-27T19:44:15Z)
• Distribution Shift Matters for Knowledge Distillation with Webly Collected Images [91.66661969598755]
  We propose a novel method dubbed Knowledge Distillation between Different Distributions" (KD$3$) We first dynamically select useful training instances from the webly collected data according to the combined predictions of teacher network and student network. We also build a new contrastive learning block called MixDistribution to generate perturbed data with a new distribution for instance alignment.
  arXiv  Detail & Related papers  (2023-07-21T10:08:58Z)
• EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation [61.33010904301476]
  Event cameras sense per-pixel intensity changes and produce asynchronous event streams with high dynamic range and less motion blur. We propose a novel approach, called bfEvDistill, to learn a student network on the unlabeled and unpaired event data. We show that EvDistill achieves significantly better results than the prior works and KD with only events and APS frames.
  arXiv  Detail & Related papers  (2021-11-24T08:48:16Z)
• Adversarial Feature Hallucination Networks for Few-Shot Learning [84.31660118264514]
  Adversarial Feature Hallucination Networks (AFHN) is based on conditional Wasserstein Generative Adversarial networks (cWGAN) Two novel regularizers are incorporated into AFHN to encourage discriminability and diversity of the synthesized features.
  arXiv  Detail & Related papers  (2020-03-30T02:43:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.