Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers

• URL: http://arxiv.org/abs/2607.27737v1
• Date: Thu, 30 Jul 2026 06:20:37 GMT
• Title: Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers
• Abstract summary: We introduce an extra teacher model (clean teacher) trained with only clean inputs, into the distillation process from a robust teacher model trained by adversarial training.<n>The features of both clean and robust teachers are transferred to the student through a cross-layer attention matrix.<n>Our methods are competitive with state-of-the-art approaches, including the recent dual-teacher distillation framework B-MTARD.
• Score: 0.34998703934432673
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks (DNNs) have achieved remarkable success in classical machine learning problems. However, they are known to be vulnerable to adversarial attacks. Countermeasures proposed in the literature, notably Information Bottleneck Distillation (IBD) introduced by Kuang et al., degrade the classification accuracy on clean inputs while improving the robustness to adversarial inputs. In this work, we extend the IBD framework by introducing an extra teacher model (clean teacher) trained with only clean inputs, into the distillation process from a robust teacher model trained by adversarial training. The features of both clean and robust teachers are transferred to the student through a cross-layer attention matrix. Experimental results on the CIFAR-10 and CIFAR-100 datasets show that the proposed method improves classification accuracy on clean samples compared to the original IBD, while maintaining similar accuracy on adversarial samples. Furthermore, our methods are competitive with state-of-the-art approaches, including the recent dual-teacher distillation framework B-MTARD, particularly in terms of the harmonic mean between clean and robust accuracy. We also analyze the impact of different training settings that have different influences on the attention module.

Related papers

• Distillation Traps and Guards: A Calibration Knob for LLM Distillability [54.90137955363471]
  We propose a post-hoc calibration method that enables control over a teacher's distillability via reinforcement fine-tuning.<n>Our objective combines task utility, KL anchor, and across-tokenizer calibration reward.<n> Experiments show that students distilled from distillable teachers outperform SFT and KD baselines.
  arXiv  Detail & Related papers  (2026-04-21T01:22:35Z)
• Deep Positive-Negative Prototypes for Adversarially Robust Discriminative Prototypical Learning [0.24999074238880484]
  We propose a novel framework named Adversarially trained Deep Positive-Negative Prototypes (Adv-DPNP)<n>Adv-DPNP integrates discriminative prototype-based learning with adversarial training.<n>We show that Adv-DPNP achieves the highest average accuracy across severities and corruption types.
  arXiv  Detail & Related papers  (2025-04-03T15:42:58Z)
• Dynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledge [17.382306203152943]
  Dynamic Guidance Adversarial Distillation (DGAD) framework tackles the challenge of differential sample importance. DGAD employs Misclassification-Aware Partitioning (MAP) to dynamically tailor the distillation focus. Error-corrective Label Swapping (ELS) corrects misclassifications of the teacher on both clean and adversarially perturbed inputs.
  arXiv  Detail & Related papers  (2024-09-03T05:52:37Z)
• Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation [12.39860047886679]
  Adversarial Training is a practical approach for improving the robustness of deep neural networks against adversarial attacks. We introduce Balanced Multi-Teacher Adversarial Robustness Distillation (B-MTARD) to guide the model's Adversarial Training process. B-MTARD outperforms the state-of-the-art methods against various adversarial attacks.
  arXiv  Detail & Related papers  (2023-06-28T12:47:01Z)
• Blending adversarial training and representation-conditional purification via aggregation improves adversarial robustness [6.484231366444063]
  CARSO is able to defend itself against adaptive end-to-end white-box attacks devised for defences.<n>Our method improves by a significant margin the state-of-the-art for Cifar-10, Cifar-100, and TinyImageNet-200.
  arXiv  Detail & Related papers  (2023-05-25T09:04:31Z)
• Robustness through Cognitive Dissociation Mitigation in Contrastive Adversarial Training [2.538209532048867]
  We introduce a novel neural network training framework that increases model's adversarial robustness to adversarial attacks. We propose to improve model robustness to adversarial attacks by learning feature representations consistent under both data augmentations and adversarial perturbations. We validate our method on the CIFAR-10 dataset on which it outperforms both robust accuracy and clean accuracy over alternative supervised and self-supervised adversarial learning methods.
  arXiv  Detail & Related papers  (2022-03-16T21:41:27Z)
• On the benefits of knowledge distillation for adversarial robustness [53.41196727255314]
  We show that knowledge distillation can be used directly to boost the performance of state-of-the-art models in adversarial robustness. We present Adversarial Knowledge Distillation (AKD), a new framework to improve a model's robust performance.
  arXiv  Detail & Related papers  (2022-03-14T15:02:13Z)
• Robust and Accurate Object Detection via Self-Knowledge Distillation [9.508466066051572]
  Unified Decoupled Feature Alignment (UDFA) is a novel fine-tuning paradigm which achieves better performance than existing methods. We show that UDFA can surpass the standard training and state-of-the-art adversarial training methods for object detection.
  arXiv  Detail & Related papers  (2021-11-14T04:40:15Z)
• How Robust are Randomized Smoothing based Defenses to Data Poisoning? [66.80663779176979]
  We present a previously unrecognized threat to robust machine learning models that highlights the importance of training-data quality. We propose a novel bilevel optimization-based data poisoning attack that degrades the robustness guarantees of certifiably robust classifiers. Our attack is effective even when the victim trains the models from scratch using state-of-the-art robust training methods.
  arXiv  Detail & Related papers  (2020-12-02T15:30:21Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)
• Adversarial Self-Supervised Contrastive Learning [62.17538130778111]
  Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions. We propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples. We present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data.
  arXiv  Detail & Related papers  (2020-06-13T08:24:33Z)
• Adversarial Robustness on In- and Out-Distribution Improves Explainability [109.68938066821246]
  RATIO is a training procedure for robustness via Adversarial Training on In- and Out-distribution. RATIO achieves state-of-the-art $l$-adrial on CIFAR10 and maintains better clean accuracy.
  arXiv  Detail & Related papers  (2020-03-20T18:57:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.