Revisiting Data-Free Knowledge Distillation with Poisoned Teachers

• URL: http://arxiv.org/abs/2306.02368v1
• Date: Sun, 4 Jun 2023 14:27:50 GMT
• Title: Revisiting Data-Free Knowledge Distillation with Poisoned Teachers
• Authors: Junyuan Hong, Yi Zeng, Shuyang Yu, Lingjuan Lyu, Ruoxi Jia, Jiayu Zhou
• Abstract summary: Data-free knowledge distillation (KD) helps transfer knowledge from a pre-trained model to a smaller model (known as the student model) without access to the original training data used for training the teacher model. However, the security of the synthetic or out-of-distribution (OOD) data required in data-free KD is largely unknown and under-explored. We propose Anti-Backdoor Data-Free KD, the first plug-in defensive method for data-free KD methods to mitigate the chance of potential backdoors being transferred.
• Score: 47.513721590643435
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data-free knowledge distillation (KD) helps transfer knowledge from a pre-trained model (known as the teacher model) to a smaller model (known as the student model) without access to the original training data used for training the teacher model. However, the security of the synthetic or out-of-distribution (OOD) data required in data-free KD is largely unknown and under-explored. In this work, we make the first effort to uncover the security risk of data-free KD w.r.t. untrusted pre-trained models. We then propose Anti-Backdoor Data-Free KD (ABD), the first plug-in defensive method for data-free KD methods to mitigate the chance of potential backdoors being transferred. We empirically evaluate the effectiveness of our proposed ABD in diminishing transferred backdoor knowledge while maintaining compatible downstream performances as the vanilla KD. We envision this work as a milestone for alarming and mitigating the potential backdoors in data-free KD. Codes are released at https://github.com/illidanlab/ABD.

Related papers

• Condensed Sample-Guided Model Inversion for Knowledge Distillation [42.91823325342862]
  Knowledge distillation (KD) is a key element in neural network compression that allows knowledge transfer from a pre-trained teacher model to a more compact student model. KD relies on access to the training dataset, which may not always be fully available due to privacy concerns or logistical issues related to the size of the data. In this paper, we consider condensed samples as a form of supplementary information, and introduce a method for using them to better approximate the target data distribution.
  arXiv  Detail & Related papers  (2024-08-25T14:43:27Z)
• Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL Shader Images [5.439020425819001]
  Knowledge distillation (KD) has been a popular and effective method for model compression. "Data-free" KD has emerged as a growing research topic which focuses on the scenario of performing KD when no data is provided. We propose a new approach to data-free KD that utilizes unnatural images, combined with large amounts of data augmentation and adversarial attacks.
  arXiv  Detail & Related papers  (2023-10-20T19:28:50Z)
• Swing Distillation: A Privacy-Preserving Knowledge Distillation Framework [38.68736962054861]
  We propose a novel knowledge distillation method, which can effectively protect the private information of the teacher model from flowing to the student model. Experiments on multiple datasets and tasks demonstrate that the proposed swing distillation can significantly reduce (by over 80% in terms of canary exposure) the risk of privacy leakage.
  arXiv  Detail & Related papers  (2022-12-16T08:57:18Z)
• Unbiased Knowledge Distillation for Recommendation [66.82575287129728]
  Knowledge distillation (KD) has been applied in recommender systems (RS) to reduce inference latency. Traditional solutions first train a full teacher model from the training data, and then transfer its knowledge to supervise the learning of a compact student model. We find such a standard distillation paradigm would incur serious bias issue -- popular items are more heavily recommended after the distillation.
  arXiv  Detail & Related papers  (2022-11-27T05:14:03Z)
• Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation [66.25738680429463]
  Knowledge Distillation (KD) for object detection aims to train a compact detector by transferring knowledge from a teacher model. We propose inconsistent knowledge distillation (IKD) which aims to distill knowledge inherent in the teacher model's counter-intuitive perceptions. Our method outperforms state-of-the-art KD baselines on one-stage, two-stage and anchor-free object detectors.
  arXiv  Detail & Related papers  (2022-09-20T16:36:28Z)
• Safe Distillation Box [62.32105311993915]
  We propose a novel framework, termed as Safe Distillation Box (SDB), that allows us to wrap a pre-trained model in a virtual box for intellectual property protection. SDB preserves the inference capability of the wrapped model to all users, but precludes KD from unauthorized users. For authorized users, on the other hand, SDB carries out a knowledge augmentation scheme to strengthen the KD performances and the results of the student model.
  arXiv  Detail & Related papers  (2021-12-05T05:01:55Z)
• Undistillable: Making A Nasty Teacher That CANNOT teach students [84.6111281091602]
  This paper introduces and investigates a concept called Nasty Teacher: a specially trained teacher network that yields nearly the same performance as a normal one. We propose a simple yet effective algorithm to build the nasty teacher, called self-undermining knowledge distillation.
  arXiv  Detail & Related papers  (2021-05-16T08:41:30Z)
• Knowledge Distillation Thrives on Data Augmentation [65.58705111863814]
  Knowledge distillation (KD) is a general deep neural network training framework that uses a teacher model to guide a student model. Many works have explored the rationale for its success, however, its interplay with data augmentation (DA) has not been well recognized so far. In this paper, we are motivated by an interesting observation in classification: KD loss can benefit from extended training iterations while the cross-entropy loss does not. We show this disparity arises because of data augmentation: KD loss can tap into the extra information from different input views brought by DA.
  arXiv  Detail & Related papers  (2020-12-05T00:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.