MIMIR: Masked Image Modeling for Mutual Information-based Adversarial
Robustness
- URL: http://arxiv.org/abs/2312.04960v2
- Date: Wed, 17 Jan 2024 13:47:32 GMT
- Title: MIMIR: Masked Image Modeling for Mutual Information-based Adversarial
Robustness
- Authors: Xiaoyun Xu, Shujian Yu, Jingzheng Wu, Stjepan Picek
- Abstract summary: Vision Transformers (ViTs) achieve superior performance on various tasks compared to convolutional neural networks (CNNs)
This paper proposes a novel defense method, MIMIR, which aims to build a different adversarial training methodology by utilizing Masked Image Modeling at pre-training.
Our results show that MIMIR improves (natural and adversarial) accuracy on average by 4.19% on CIFAR-10 and 5.52% on ImageNet-1K, compared to baselines.
- Score: 31.76309077313509
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Vision Transformers (ViTs) achieve superior performance on various tasks
compared to convolutional neural networks (CNNs), but ViTs are also vulnerable
to adversarial attacks. Adversarial training is one of the most successful
methods to build robust CNN models. Thus, recent works explored new
methodologies for adversarial training of ViTs based on the differences between
ViTs and CNNs, such as better training strategies, preventing attention from
focusing on a single block, or discarding low-attention embeddings. However,
these methods still follow the design of traditional supervised adversarial
training, limiting the potential of adversarial training on ViTs. This paper
proposes a novel defense method, MIMIR, which aims to build a different
adversarial training methodology by utilizing Masked Image Modeling at
pre-training. We create an autoencoder that accepts adversarial examples as
input but takes the clean examples as the modeling target. Then, we create a
mutual information (MI) penalty following the idea of the Information
Bottleneck. Among the two information source inputs and corresponding
adversarial perturbation, the perturbation information is eliminated due to the
constraint of the modeling target. Next, we provide a theoretical analysis of
MIMIR using the bounds of the MI penalty. We also design two adaptive attacks
when the adversary is aware of the MIMIR defense and show that MIMIR still
performs well. The experimental results show that MIMIR improves (natural and
adversarial) accuracy on average by 4.19% on CIFAR-10 and 5.52% on ImageNet-1K,
compared to baselines. On Tiny-ImageNet, we obtained improved natural accuracy
of 2.99\% on average and comparable adversarial accuracy. Our code and trained
models are publicly available https://github.com/xiaoyunxxy/MIMIR.
Related papers
- Fast Propagation is Better: Accelerating Single-Step Adversarial
Training via Sampling Subnetworks [69.54774045493227]
A drawback of adversarial training is the computational overhead introduced by the generation of adversarial examples.
We propose to exploit the interior building blocks of the model to improve efficiency.
Compared with previous methods, our method not only reduces the training cost but also achieves better model robustness.
arXiv Detail & Related papers (2023-10-24T01:36:20Z) - Outlier Robust Adversarial Training [57.06824365801612]
We introduce Outlier Robust Adversarial Training (ORAT) in this work.
ORAT is based on a bi-level optimization formulation of adversarial training with a robust rank-based loss function.
We show that the learning objective of ORAT satisfies the $mathcalH$-consistency in binary classification, which establishes it as a proper surrogate to adversarial 0/1 loss.
arXiv Detail & Related papers (2023-09-10T21:36:38Z) - Masking and Mixing Adversarial Training [9.690454593095495]
Adversarial training is a popular and straightforward technique to defend against the threat of adversarial examples.
CNNs must sacrifice the accuracy of standard samples to improve robustness against adversarial examples.
We propose Masking and Mixing Adversarial Training (M2AT) to mitigate the trade-off between accuracy and robustness.
arXiv Detail & Related papers (2023-02-16T04:05:53Z) - Beyond Pretrained Features: Noisy Image Modeling Provides Adversarial
Defense [52.66971714830943]
Masked image modeling (MIM) has made it a prevailing framework for self-supervised visual representation learning.
In this paper, we investigate how this powerful self-supervised learning paradigm can provide adversarial robustness to downstream classifiers.
We propose an adversarial defense method, referred to as De3, by exploiting the pretrained decoder for denoising.
arXiv Detail & Related papers (2023-02-02T12:37:24Z) - Bayesian Learning with Information Gain Provably Bounds Risk for a
Robust Adversarial Defense [27.545466364906773]
We present a new algorithm to learn a deep neural network model robust against adversarial attacks.
Our model demonstrate significantly improved robustness--up to 20%--compared with adversarial training and Adv-BNN under PGD attacks.
arXiv Detail & Related papers (2022-12-05T03:26:08Z) - AccelAT: A Framework for Accelerating the Adversarial Training of Deep
Neural Networks through Accuracy Gradient [12.118084418840152]
Adrial training is exploited to develop a robust Deep Neural Network (DNN) model against malicious altered data.
This paper aims at accelerating the adversarial training to enable fast development of robust DNN models against adversarial attacks.
arXiv Detail & Related papers (2022-10-13T10:31:51Z) - Two Heads are Better than One: Robust Learning Meets Multi-branch Models [14.72099568017039]
We propose Branch Orthogonality adveRsarial Training (BORT) to obtain state-of-the-art performance with solely the original dataset for adversarial training.
We evaluate our approach on CIFAR-10, CIFAR-100, and SVHN against ell_infty norm-bounded perturbations of size epsilon = 8/255, respectively.
arXiv Detail & Related papers (2022-08-17T05:42:59Z) - RelaxLoss: Defending Membership Inference Attacks without Losing Utility [68.48117818874155]
We propose a novel training framework based on a relaxed loss with a more achievable learning target.
RelaxLoss is applicable to any classification model with added benefits of easy implementation and negligible overhead.
Our approach consistently outperforms state-of-the-art defense mechanisms in terms of resilience against MIAs.
arXiv Detail & Related papers (2022-07-12T19:34:47Z) - Understanding the Logit Distributions of Adversarially-Trained Deep
Neural Networks [6.439477789066243]
Adversarial defenses train deep neural networks to be invariant to the input perturbations from adversarial attacks.
Although adversarial training is successful at mitigating adversarial attacks, the behavioral differences between adversarially-trained (AT) models and standard models are still poorly understood.
We identify three logit characteristics essential to learning adversarial robustness.
arXiv Detail & Related papers (2021-08-26T19:09:15Z) - Stylized Adversarial Defense [105.88250594033053]
adversarial training creates perturbation patterns and includes them in the training set to robustify the model.
We propose to exploit additional information from the feature space to craft stronger adversaries.
Our adversarial training approach demonstrates strong robustness compared to state-of-the-art defenses.
arXiv Detail & Related papers (2020-07-29T08:38:10Z) - Towards Achieving Adversarial Robustness by Enforcing Feature
Consistency Across Bit Planes [51.31334977346847]
We train networks to form coarse impressions based on the information in higher bit planes, and use the lower bit planes only to refine their prediction.
We demonstrate that, by imposing consistency on the representations learned across differently quantized images, the adversarial robustness of networks improves significantly.
arXiv Detail & Related papers (2020-04-01T09:31:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.