Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation

• URL: http://arxiv.org/abs/2507.20996v1
• Date: Mon, 28 Jul 2025 17:08:40 GMT
• Title: Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation
• Authors: Hayat Ullah, Syed Muhammad Talha Zaidi, Arslan Munir,
• Abstract summary: Convolutional neural networks (CNNs) excel in computer vision but are susceptible to adversarial attacks.<n>Despite advances in adversarial training, a gap persists between model accuracy and robustness.<n>We present a multi-teacher adversarial robustness distillation using an adaptive learning strategy.
• Score: 1.8609604872307923
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Convolutional neural networks (CNNs) excel in computer vision but are susceptible to adversarial attacks, crafted perturbations designed to mislead predictions. Despite advances in adversarial training, a gap persists between model accuracy and robustness. To mitigate this issue, in this paper, we present a multi-teacher adversarial robustness distillation using an adaptive learning strategy. Specifically, our proposed method first trained multiple clones of a baseline CNN model using an adversarial training strategy on a pool of perturbed data acquired through different adversarial attacks. Once trained, these adversarially trained models are used as teacher models to supervise the learning of a student model on clean data using multi-teacher knowledge distillation. To ensure an effective robustness distillation, we design an adaptive learning strategy that controls the knowledge contribution of each model by assigning weights as per their prediction precision. Distilling knowledge from adversarially pre-trained teacher models not only enhances the learning capabilities of the student model but also empowers it with the capacity to withstand different adversarial attacks, despite having no exposure to adversarial data. To verify our claims, we extensively evaluated our proposed method on MNIST-Digits and Fashion-MNIST datasets across diverse experimental settings. The obtained results exhibit the efficacy of our multi-teacher adversarial distillation and adaptive learning strategy, enhancing CNNs' adversarial robustness against various adversarial attacks.

Related papers

• Sustainable Self-evolution Adversarial Training [51.25767996364584]
  We propose a Sustainable Self-Evolution Adversarial Training (SSEAT) framework for adversarial training defense models.<n>We introduce a continual adversarial defense pipeline to realize learning from various kinds of adversarial examples.<n>We also propose an adversarial data replay module to better select more diverse and key relearning data.
  arXiv  Detail & Related papers  (2024-12-03T08:41:11Z)
• Enhancing Adversarial Robustness through Multi-Objective Representation Learning [1.534667887016089]
  Deep neural networks (DNNs) are vulnerable to small adversarial perturbations.<n>We show that robust feature learning during training can significantly enhance robustness.<n>We propose MOREL, a multi-objective approach that aligns natural and adversarial features.
  arXiv  Detail & Related papers  (2024-10-02T16:05:03Z)
• Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation [12.39860047886679]
  Adversarial Training is a practical approach for improving the robustness of deep neural networks against adversarial attacks. We introduce Balanced Multi-Teacher Adversarial Robustness Distillation (B-MTARD) to guide the model's Adversarial Training process. B-MTARD outperforms the state-of-the-art methods against various adversarial attacks.
  arXiv  Detail & Related papers  (2023-06-28T12:47:01Z)
• Bayesian Learning with Information Gain Provably Bounds Risk for a Robust Adversarial Defense [27.545466364906773]
  We present a new algorithm to learn a deep neural network model robust against adversarial attacks. Our model demonstrate significantly improved robustness--up to 20%--compared with adversarial training and Adv-BNN under PGD attacks.
  arXiv  Detail & Related papers  (2022-12-05T03:26:08Z)
• Self-Ensemble Adversarial Training for Improved Robustness [14.244311026737666]
  Adversarial training is the strongest strategy against various adversarial attacks among all sorts of defense methods. Recent works mainly focus on developing new loss functions or regularizers, attempting to find the unique optimal point in the weight space. We devise a simple but powerful emphSelf-Ensemble Adversarial Training (SEAT) method for yielding a robust classifier by averaging weights of history models.
  arXiv  Detail & Related papers  (2022-03-18T01:12:18Z)
• Learning to Learn Transferable Attack [77.67399621530052]
  Transfer adversarial attack is a non-trivial black-box adversarial attack that aims to craft adversarial perturbations on the surrogate model and then apply such perturbations to the victim model. We propose a Learning to Learn Transferable Attack (LLTA) method, which makes the adversarial perturbations more generalized via learning from both data and model augmentation. Empirical results on the widely-used dataset demonstrate the effectiveness of our attack method with a 12.85% higher success rate of transfer attack compared with the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-12-10T07:24:21Z)
• Mutual Adversarial Training: Learning together is better than going alone [82.78852509965547]
  We study how interactions among models affect robustness via knowledge distillation. We propose mutual adversarial training (MAT) in which multiple models are trained together. MAT can effectively improve model robustness and outperform state-of-the-art methods under white-box attacks.
  arXiv  Detail & Related papers  (2021-12-09T15:59:42Z)
• Understanding the Logit Distributions of Adversarially-Trained Deep Neural Networks [6.439477789066243]
  Adversarial defenses train deep neural networks to be invariant to the input perturbations from adversarial attacks. Although adversarial training is successful at mitigating adversarial attacks, the behavioral differences between adversarially-trained (AT) models and standard models are still poorly understood. We identify three logit characteristics essential to learning adversarial robustness.
  arXiv  Detail & Related papers  (2021-08-26T19:09:15Z)
• Delving into Data: Effectively Substitute Training for Black-box Attack [84.85798059317963]
  We propose a novel perspective substitute training that focuses on designing the distribution of data used in the knowledge stealing process. The combination of these two modules can further boost the consistency of the substitute model and target model, which greatly improves the effectiveness of adversarial attack.
  arXiv  Detail & Related papers  (2021-04-26T07:26:29Z)
• Feature Distillation With Guided Adversarial Contrastive Learning [41.28710294669751]
  We propose Guided Adversarial Contrastive Distillation (GACD) to transfer adversarial robustness from teacher to student with features. With a well-trained teacher model as an anchor, students are expected to extract features similar to the teacher. With GACD, the student not only learns to extract robust features, but also captures structural knowledge from the teacher.
  arXiv  Detail & Related papers  (2020-09-21T14:46:17Z)
• Stylized Adversarial Defense [105.88250594033053]
  adversarial training creates perturbation patterns and includes them in the training set to robustify the model. We propose to exploit additional information from the feature space to craft stronger adversaries. Our adversarial training approach demonstrates strong robustness compared to state-of-the-art defenses.
  arXiv  Detail & Related papers  (2020-07-29T08:38:10Z)
• Adversarial Self-Supervised Contrastive Learning [62.17538130778111]
  Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions. We propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples. We present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data.
  arXiv  Detail & Related papers  (2020-06-13T08:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.