InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle

• URL: http://arxiv.org/abs/2206.12292v1
• Date: Thu, 23 Jun 2022 03:20:41 GMT
• Title: InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle
• Abstract summary: Adversarial training (AT) has shown excellent high performance in defending against adversarial examples. We propose a novel and effective adversarial training method (InfoAT) Experimental results show that InfoAT achieves the best robustness among different datasets and models in comparison with several state-of-the-art methods.
• Score: 22.531976474053057
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adversarial training (AT) has shown excellent high performance in defending against adversarial examples. Recent studies demonstrate that examples are not equally important to the final robustness of models during AT, that is, the so-called hard examples that can be attacked easily exhibit more influence than robust examples on the final robustness. Therefore, guaranteeing the robustness of hard examples is crucial for improving the final robustness of the model. However, defining effective heuristics to search for hard examples is still difficult. In this article, inspired by the information bottleneck (IB) principle, we uncover that an example with high mutual information of the input and its associated latent representation is more likely to be attacked. Based on this observation, we propose a novel and effective adversarial training method (InfoAT). InfoAT is encouraged to find examples with high mutual information and exploit them efficiently to improve the final robustness of models. Experimental results show that InfoAT achieves the best robustness among different datasets and models in comparison with several state-of-the-art methods.

Related papers

• What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection [25.414327410290543]
  On-Policy Distillation (OPD) has emerged as a widely adopted post-training paradigm for enhancing large language models in reasoning domains.<n>This paper presents a empirical study of data efficiency and data selection in OPD.
  arXiv  Detail & Related papers  (2026-09-04T14:32:08Z)
• A Validation Strategy for Deep Learning Models: Evaluating and Enhancing Robustness [0.8532585403388676]
  We propose a validation approach that extracts "weak robust" samples directly from the training dataset via local analysis.<n>These samples, being the most susceptible to perturbations, serve as an early and sensitive indicator of the model's vulnerabilities.<n>We demonstrate the effectiveness of our approach on models trained with CIFAR-10, CIFAR-100, and ImageNet.
  arXiv  Detail & Related papers  (2025-09-23T16:14:14Z)
• DARD: Dice Adversarial Robustness Distillation against Adversarial Attacks [12.90150211072263]
  We introduce Dice Adversarial Robustness Distillation (DARD), a novel method designed to transfer robustness through a tailored knowledge distillation paradigm.<n>Our experiments demonstrate that the DARD approach consistently outperforms adversarially trained networks with the same architecture.
  arXiv  Detail & Related papers  (2025-09-15T02:31:30Z)
• Towards Model Resistant to Transferable Adversarial Examples via Trigger Activation [95.3977252782181]
  Adversarial examples, characterized by imperceptible perturbations, pose significant threats to deep neural networks by misleading their predictions. We introduce a novel training paradigm aimed at enhancing robustness against transferable adversarial examples (TAEs) in a more efficient and effective way.
  arXiv  Detail & Related papers  (2025-04-20T09:07:10Z)
• OMG-ATTACK: Self-Supervised On-Manifold Generation of Transferable Evasion Attacks [17.584752814352502]
  Evasion Attacks (EA) are used to test the robustness of trained neural networks by distorting input data. We introduce a self-supervised, computationally economical method for generating adversarial examples. Our experiments consistently demonstrate the method is effective across various models, unseen data categories, and even defended models.
  arXiv  Detail & Related papers  (2023-10-05T17:34:47Z)
• Doubly Robust Instance-Reweighted Adversarial Training [107.40683655362285]
  We propose a novel doubly-robust instance reweighted adversarial framework. Our importance weights are obtained by optimizing the KL-divergence regularized loss function. Our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance.
  arXiv  Detail & Related papers  (2023-08-01T06:16:18Z)
• Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck [77.37409441129995]
  In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition. We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
  arXiv  Detail & Related papers  (2023-03-24T16:03:21Z)
• The Enemy of My Enemy is My Friend: Exploring Inverse Adversaries for Improving Adversarial Training [72.39526433794707]
  Adversarial training and its variants have been shown to be the most effective approaches to defend against adversarial examples. We propose a novel adversarial training scheme that encourages the model to produce similar outputs for an adversarial example and its inverse adversarial'' counterpart. Our training method achieves state-of-the-art robustness as well as natural accuracy.
  arXiv  Detail & Related papers  (2022-11-01T15:24:26Z)
• Improving the Adversarial Robustness of NLP Models by Information Bottleneck [112.44039792098579]
  Non-robust features can be easily manipulated by adversaries to fool NLP models. In this study, we explore the feasibility of capturing task-specific robust features, while eliminating the non-robust ones by using the information bottleneck theory. We show that the models trained with our information bottleneck-based method are able to achieve a significant improvement in robust accuracy.
  arXiv  Detail & Related papers  (2022-06-11T12:12:20Z)
• Adversarial Examples for Good: Adversarial Examples Guided Imbalanced Learning [15.370413523189749]
  We provide a new perspective on how to deal with imbalanced data: adjust the biased decision boundary by training with Guiding Adversarial Examples (GAEs) Our method can effectively increase the accuracy of minority classes while sacrificing little accuracy on majority classes.
  arXiv  Detail & Related papers  (2022-01-28T09:13:07Z)
• Mutual Adversarial Training: Learning together is better than going alone [82.78852509965547]
  We study how interactions among models affect robustness via knowledge distillation. We propose mutual adversarial training (MAT) in which multiple models are trained together. MAT can effectively improve model robustness and outperform state-of-the-art methods under white-box attacks.
  arXiv  Detail & Related papers  (2021-12-09T15:59:42Z)
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning [25.85044477227461]
  Models that are more accurate on the out-of-distribution data relative to this baseline exhibit "effective robustness" We find that models pre-trained on larger datasets exhibit effective robustness during training that vanishes at convergence. We discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models.
  arXiv  Detail & Related papers  (2021-06-30T06:21:42Z)
• MixKD: Towards Efficient Distillation of Large-scale Language Models [129.73786264834894]
  We propose MixKD, a data-agnostic distillation framework, to endow the resulting model with stronger generalization ability. We prove from a theoretical perspective that under reasonable conditions MixKD gives rise to a smaller gap between the error and the empirical error. Experiments under a limited-data setting and ablation studies further demonstrate the advantages of the proposed approach.
  arXiv  Detail & Related papers  (2020-11-01T18:47:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.