When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need

• URL: http://arxiv.org/abs/2507.04119v1
• Date: Sat, 05 Jul 2025 18:03:52 GMT
• Title: When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need
• Abstract summary: Data-free knowledge distillation (DFKD) transfers knowledge from a teacher to a student without access to the real in-distribution (ID) data.<n>Existing works typically assume teachers are trustworthy, leaving the robustness and security of DFKD from untrusted teachers largely unexplored.<n>We conduct the first investigation into distilling non-transferable learning (NTL) teachers using DFKD.<n>We find that NTL teachers fool DFKD through divert the generator's attention from the useful ID knowledge to the misleading OOD knowledge.
• Score: 78.71039931096959
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data-free knowledge distillation (DFKD) transfers knowledge from a teacher to a student without access the real in-distribution (ID) data. Its common solution is to use a generator to synthesize fake data and use them as a substitute for real ID data. However, existing works typically assume teachers are trustworthy, leaving the robustness and security of DFKD from untrusted teachers largely unexplored. In this work, we conduct the first investigation into distilling non-transferable learning (NTL) teachers using DFKD, where the transferability from an ID domain to an out-of-distribution (OOD) domain is prohibited. We find that NTL teachers fool DFKD through divert the generator's attention from the useful ID knowledge to the misleading OOD knowledge. This hinders ID knowledge transfer but prioritizes OOD knowledge transfer. To mitigate this issue, we propose Adversarial Trap Escaping (ATEsc) to benefit DFKD by identifying and filtering out OOD-like synthetic samples. Specifically, inspired by the evidence that NTL teachers show stronger adversarial robustness on OOD samples than ID samples, we split synthetic samples into two groups according to their robustness. The fragile group is treated as ID-like data and used for normal knowledge distillation, while the robust group is seen as OOD-like data and utilized for forgetting OOD knowledge. Extensive experiments demonstrate the effectiveness of ATEsc for improving DFKD against NTL teachers. Code is released at https://github.com/tmllab/2025_ICML_ATEsc.

Related papers

• LoD: Loss-difference OOD Detection by Intentionally Label-Noisifying Unlabeled Wild Data [45.32174880762807]
  We propose a novel loss-difference OOD detection framework (LoD) by textitintentionally label-noisifying unlabeled wild data.<n>Such operations not only enable labeled ID data and OOD data in unlabeled wild data to jointly dominate the models' learning but also ensure the distinguishability of the losses between ID and OOD samples in unlabeled wild data.
  arXiv  Detail & Related papers  (2025-05-19T10:44:52Z)
• AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation [33.208860361882095]
  Data-Free Knowledge Distillation (DFKD) methods have emerged as direct solutions. However, simply adopting models derived from DFKD for real-world applications suffers significant performance degradation. We propose a simple but effective method AuG-KD to selectively transfer teachers' appropriate knowledge.
  arXiv  Detail & Related papers  (2024-03-11T03:34:14Z)
• How Does Unlabeled Data Provably Help Out-of-Distribution Detection? [63.41681272937562]
  Unlabeled in-the-wild data is non-trivial due to the heterogeneity of both in-distribution (ID) and out-of-distribution (OOD) data. This paper introduces a new learning framework SAL (Separate And Learn) that offers both strong theoretical guarantees and empirical effectiveness.
  arXiv  Detail & Related papers  (2024-02-05T20:36:33Z)
• Out-of-distribution Detection Learning with Unreliable Out-of-distribution Sources [73.28967478098107]
  Out-of-distribution (OOD) detection discerns OOD data where the predictor cannot make valid predictions as in-distribution (ID) data. It is typically hard to collect real out-of-distribution (OOD) data for training a predictor capable of discerning OOD patterns. We propose a data generation-based learning method named Auxiliary Task-based OOD Learning (ATOL) that can relieve the mistaken OOD generation.
  arXiv  Detail & Related papers  (2023-11-06T16:26:52Z)
• Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation [66.25738680429463]
  Knowledge Distillation (KD) for object detection aims to train a compact detector by transferring knowledge from a teacher model. We propose inconsistent knowledge distillation (IKD) which aims to distill knowledge inherent in the teacher model's counter-intuitive perceptions. Our method outperforms state-of-the-art KD baselines on one-stage, two-stage and anchor-free object detectors.
  arXiv  Detail & Related papers  (2022-09-20T16:36:28Z)
• Semi-Online Knowledge Distillation [2.373824287636486]
  Conventional knowledge distillation (KD) is to transfer knowledge from a large and well pre-trained teacher network to a small student network. Deep mutual learning (DML) has been proposed to help student networks learn collaboratively and simultaneously. We propose a Semi-Online Knowledge Distillation (SOKD) method that effectively improves the performance of the student and the teacher.
  arXiv  Detail & Related papers  (2021-11-23T09:44:58Z)
• Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data [56.29595334715237]
  Knowledge distillation(KD) aims to craft a compact student model that imitates the behavior of a pre-trained teacher in a target domain. We introduce a handy yet surprisingly efficacious approach, dubbed astextitMosaicKD. In MosaicKD, this is achieved through a four-player min-max game, in which a generator, a discriminator, a student network, are collectively trained in an adversarial manner.
  arXiv  Detail & Related papers  (2021-10-27T13:01:10Z)
• They are Not Completely Useless: Towards Recycling Transferable Unlabeled Data for Class-Mismatched Semi-Supervised Learning [61.46572463531167]
  Semi-Supervised Learning (SSL) with mismatched classes deals with the problem that the classes-of-interests in the limited labeled data is only a subset of the classes in massive unlabeled data. This paper proposes a "Transferable OOD data Recycling" (TOOR) method to enrich the information for conducting class-mismatched SSL.
  arXiv  Detail & Related papers  (2020-11-27T02:29:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.