Improving Robustness using Generated Data

• URL: http://arxiv.org/abs/2110.09468v1
• Date: Mon, 18 Oct 2021 17:00:26 GMT
• Title: Improving Robustness using Generated Data
• Authors: Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, Timothy Mann
• Abstract summary: generative models trained solely on the original training set can be leveraged to artificially increase the size of the original training set. We show large absolute improvements in robust accuracy compared to previous state-of-the-art methods.
• Score: 20.873767830152605
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work argues that robust training requires substantially larger datasets than those required for standard classification. On CIFAR-10 and CIFAR-100, this translates into a sizable robust-accuracy gap between models trained solely on data from the original training set and those trained with additional data extracted from the "80 Million Tiny Images" dataset (TI-80M). In this paper, we explore how generative models trained solely on the original training set can be leveraged to artificially increase the size of the original training set and improve adversarial robustness to $\ell_p$ norm-bounded perturbations. We identify the sufficient conditions under which incorporating additional generated data can improve robustness, and demonstrate that it is possible to significantly reduce the robust-accuracy gap to models trained with additional real data. Surprisingly, we even show that even the addition of non-realistic random data (generated by Gaussian sampling) can improve robustness. We evaluate our approach on CIFAR-10, CIFAR-100, SVHN and TinyImageNet against $\ell_\infty$ and $\ell_2$ norm-bounded perturbations of size $\epsilon = 8/255$ and $\epsilon = 128/255$, respectively. We show large absolute improvements in robust accuracy compared to previous state-of-the-art methods. Against $\ell_\infty$ norm-bounded perturbations of size $\epsilon = 8/255$, our models achieve 66.10% and 33.49% robust accuracy on CIFAR-10 and CIFAR-100, respectively (improving upon the state-of-the-art by +8.96% and +3.29%). Against $\ell_2$ norm-bounded perturbations of size $\epsilon = 128/255$, our model achieves 78.31% on CIFAR-10 (+3.81%). These results beat most prior works that use external data.

Related papers

• MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers [41.56951365163419]
  "MixedNUTS" is a training-free method where the output logits of a robust classifier are processed by nonlinear transformations with only three parameters. MixedNUTS then converts the transformed logits into probabilities and mixes them as the overall output. On CIFAR-10, CIFAR-100, and ImageNet datasets, experimental results with custom strong adaptive attacks demonstrate MixedNUTS's vastly improved accuracy and near-SOTA robustness.
  arXiv  Detail & Related papers  (2024-02-03T21:12:36Z)
• Better Diffusion Models Further Improve Adversarial Training [97.44991845907708]
  It has been recognized that the data generated by the diffusion probabilistic model (DDPM) improves adversarial training. This paper gives an affirmative answer by employing the most recent diffusion model which has higher efficiency. Our adversarially trained models achieve state-of-the-art performance on RobustBench using only generated data.
  arXiv  Detail & Related papers  (2023-02-09T13:46:42Z)
• Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space [51.62131362670815]
  This paper addresses the problem of ranking the pre-trained deep neural networks and screening the most transferable ones for downstream tasks. It proposes a new transferability metric called textbfSelf-challenging textbfFisher textbfDiscriminant textbfAnalysis (textbfSFDA)
  arXiv  Detail & Related papers  (2022-07-07T01:33:25Z)
• Removing Batch Normalization Boosts Adversarial Training [83.08844497295148]
  Adversarial training (AT) defends deep neural networks against adversarial attacks. A major bottleneck is the widely used batch normalization (BN), which struggles to model the different statistics of clean and adversarial training samples in AT. Our normalizer-free robust training (NoFrost) method extends recent advances in normalizer-free networks to AT.
  arXiv  Detail & Related papers  (2022-07-04T01:39:37Z)
• Data Augmentation Can Improve Robustness [21.485435979018256]
  Adrial training suffers from robust overfitting, a phenomenon where the robust test accuracy starts to decrease during training. We demonstrate that, when combined with model weight averaging, data augmentation can significantly boost robust accuracy. In particular, against $ell_infty$ norm-bounded perturbations of size $epsilon = 8/255$, our model reaches 60.07% robust accuracy without using any external data.
  arXiv  Detail & Related papers  (2021-11-09T18:57:00Z)
• Fixing Data Augmentation to Improve Adversarial Robustness [21.485435979018256]
  Adversarial training suffers from robust overfitting, a phenomenon where the robust test accuracy starts to decrease during training. In this paper, we focus on both adversarials-driven and data-driven augmentations as a means to reduce robust overfitting. We show large absolute improvements of +7.06% and +5.88% in robust accuracy compared to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-02T18:58:33Z)
• Learnable Boundary Guided Adversarial Training [66.57846365425598]
  We use the model logits from one clean model to guide learning of another one robust model. We achieve new state-of-the-art robustness on CIFAR-100 without additional real or synthetic data.
  arXiv  Detail & Related papers  (2020-11-23T01:36:05Z)
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples [47.27255244183513]
  We study the effect of different training losses, model sizes, activation functions, the addition of unlabeled data (through pseudo-labeling) and other factors on adversarial robustness. We discover that it is possible to train robust models that go well beyond state-of-the-art results by combining larger models, Swish/SiLU activations and model weight averaging.
  arXiv  Detail & Related papers  (2020-10-07T18:19:09Z)
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning [134.15174177472807]
  We introduce adversarial training into self-supervision, to provide general-purpose robust pre-trained models for the first time. We conduct extensive experiments to demonstrate that the proposed framework achieves large performance margins.
  arXiv  Detail & Related papers  (2020-03-28T18:28:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.