Revisiting adapters with adversarial training
- URL: http://arxiv.org/abs/2210.04886v1
- Date: Mon, 10 Oct 2022 17:58:14 GMT
- Title: Revisiting adapters with adversarial training
- Abstract summary: We show that it is not necessary to separate batch statistics when co-training on clean and adversarial inputs.
We show that training with adapters enables model soups through linear combinations of the clean and adversarial tokens.
- Score: 39.018061382248085
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While adversarial training is generally used as a defense mechanism, recent
works show that it can also act as a regularizer. By co-training a neural
network on clean and adversarial inputs, it is possible to improve
classification accuracy on the clean, non-adversarial inputs. We demonstrate
that, contrary to previous findings, it is not necessary to separate batch
statistics when co-training on clean and adversarial inputs, and that it is
sufficient to use adapters with few domain-specific parameters for each type of
input. We establish that using the classification token of a Vision Transformer
(ViT) as an adapter is enough to match the classification performance of dual
normalization layers, while using significantly less additional parameters.
First, we improve upon the top-1 accuracy of a non-adversarially trained
ViT-B16 model by +1.12% on ImageNet (reaching 83.76% top-1 accuracy). Second,
and more importantly, we show that training with adapters enables model soups
through linear combinations of the clean and adversarial tokens. These model
soups, which we call adversarial model soups, allow us to trade-off between
clean and robust accuracy without sacrificing efficiency. Finally, we show that
we can easily adapt the resulting models in the face of distribution shifts.
Our ViT-B16 obtains top-1 accuracies on ImageNet variants that are on average
+4.00% better than those obtained with Masked Autoencoders.
Related papers
- AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens [6.989655731780402]
We present AdaPerceiver, the first transformer architecture with unified adaptivity across depth, width, and tokens within a single model.<n>AdaPerceiver achieves 85.4% accuracy while yielding 36% higher throughput than FlexiViT-L.
arXiv Detail & Related papers (2025-11-22T16:09:14Z) - Improving Robustness of Foundation Models in Domain Adaptation with Soup-Adapters [0.0]
We show that by training multiple independent adapters and averaging their outputs, the new model has a higher performance and is more robust to distribution shifts compared to any individual adapter.<n>This is also the first study to explore CLIP adapter-style techniques for DINOv2 and to directly compare them with CLIP in this setting.
arXiv Detail & Related papers (2025-07-08T09:26:10Z) - Towards Robust Vision Transformer via Masked Adaptive Ensemble [23.986968861837813]
Adversarial training (AT) can help improve the robustness of Vision Transformers (ViT) against adversarial attacks.
This paper proposes a novel ViT architecture, including a detector and a classifier bridged by our newly developed adaptive ensemble.
Experimental results exhibit that our ViT architecture, on CIFAR-10, achieves the best standard accuracy and adversarial robustness of 90.3% and 49.8%, respectively.
arXiv Detail & Related papers (2024-07-22T05:28:29Z) - On adversarial training and the 1 Nearest Neighbor classifier [8.248839892711478]
We compare the performance of adversarial training to that of the simple 1 Nearest Neighbor (1NN) classifier.
Experiments with 135 different binary image classification problems taken from CIFAR10, MNIST and Fashion-MNIST.
We find that 1NN outperforms almost all of them in terms of robustness to perturbations that are only slightly different from those used during training.
arXiv Detail & Related papers (2024-04-09T13:47:37Z) - SkipViT: Speeding Up Vision Transformers with a Token-Level Skip
Connection [3.960622297616708]
We propose a method to optimize the amount of unnecessary interactions between unimportant tokens by separating and sending them through a different low-cost computational path.
Our experimental results on training ViT-small from scratch show that SkipViT is capable of effectively dropping 55% of the tokens while gaining more than 13% training throughput.
arXiv Detail & Related papers (2024-01-27T04:24:49Z) - Adapters Mixup: Mixing Parameter-Efficient Adapters to Enhance the Adversarial Robustness of Fine-tuned Pre-trained Text Classifiers [9.250758784663411]
AdpMixup combines fine-tuning through adapters and adversarial augmentation via mixup to dynamically leverage existing knowledge for robust inference.
Experiments show AdpMixup achieves the best trade-off between training efficiency and robustness under both pre-known and unknown attacks.
arXiv Detail & Related papers (2024-01-18T16:27:18Z) - Co-training $2^L$ Submodels for Visual Recognition [67.02999567435626]
Submodel co-training is a regularization method related to co-training, self-distillation and depth.
We show that submodel co-training is effective to train backbones for recognition tasks such as image classification and semantic segmentation.
arXiv Detail & Related papers (2022-12-09T14:38:09Z) - Prompt Tuning for Parameter-efficient Medical Image Segmentation [79.09285179181225]
We propose and investigate several contributions to achieve a parameter-efficient but effective adaptation for semantic segmentation on two medical imaging datasets.
We pre-train this architecture with a dedicated dense self-supervision scheme based on assignments to online generated prototypes.
We demonstrate that the resulting neural network model is able to attenuate the gap between fully fine-tuned and parameter-efficiently adapted models.
arXiv Detail & Related papers (2022-11-16T21:55:05Z) - SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained
Models [9.017387427570538]
Vision-language models such as CLIP are pretrained on large volumes of internet sourced image and text pairs.
Due to their size, fine-tuning these models on new datasets can be prohibitively expensive, both in terms of the supervision and compute required.
We present a new approach called SVL-Adapter that combines the complementary strengths of both vision-language pretraining and self-supervised representation learning.
arXiv Detail & Related papers (2022-10-07T19:35:08Z) - Removing Batch Normalization Boosts Adversarial Training [83.08844497295148]
Adversarial training (AT) defends deep neural networks against adversarial attacks.
A major bottleneck is the widely used batch normalization (BN), which struggles to model the different statistics of clean and adversarial training samples in AT.
Our normalizer-free robust training (NoFrost) method extends recent advances in normalizer-free networks to AT.
arXiv Detail & Related papers (2022-07-04T01:39:37Z) - Distributed Adversarial Training to Robustify Deep Neural Networks at
Scale [100.19539096465101]
Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification.
To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training.
We propose a large-batch adversarial training framework implemented over multiple machines.
arXiv Detail & Related papers (2022-06-13T15:39:43Z) - Corrupted Image Modeling for Self-Supervised Visual Pre-Training [103.99311611776697]
We introduce Corrupted Image Modeling (CIM) for self-supervised visual pre-training.
CIM uses an auxiliary generator with a small trainable BEiT to corrupt the input image instead of using artificial mask tokens.
After pre-training, the enhancer can be used as a high-capacity visual encoder for downstream tasks.
arXiv Detail & Related papers (2022-02-07T17:59:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.