Noise Augmentation Is All You Need For FGSM Fast Adversarial Training: Catastrophic Overfitting And Robust Overfitting Require Different Augmentation

• URL: http://arxiv.org/abs/2202.05488v1
• Date: Fri, 11 Feb 2022 07:37:55 GMT
• Title: Noise Augmentation Is All You Need For FGSM Fast Adversarial Training: Catastrophic Overfitting And Robust Overfitting Require Different Augmentation
• Abstract summary: Adversarial training (AT) and its variants are the most effective approaches for obtaining adversarially robust models. In this work, we show that single-step F GSMAT is not as hard as suggested in the past line of work: noise augmentation is all you need for (F GSM) fast AT.
• Score: 72.87038663276554
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adversarial training (AT) and its variants are the most effective approaches for obtaining adversarially robust models. A unique characteristic of AT is that an inner maximization problem needs to be solved repeatedly before the model weights can be updated, which makes the training slow. FGSM AT significantly improves its efficiency but it fails when the step size grows. The SOTA GradAlign makes FGSM AT compatible with a higher step size, however, its regularization on input gradient makes it 3 to 4 times slower than FGSM AT. Our proposed NoiseAug removes the extra computation overhead by directly regularizing on the input itself. The key contribution of this work lies in an empirical finding that single-step FGSM AT is not as hard as suggested in the past line of work: noise augmentation is all you need for (FGSM) fast AT. Towards understanding the success of our NoiseAug, we perform an extensive analysis and find that mitigating Catastrophic Overfitting (CO) and Robust Overfitting (RO) need different augmentations. Instead of more samples caused by data augmentation, we identify what makes NoiseAug effective for preventing CO might lie in its improved local linearity.

Related papers

• From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons [74.25767873168745]
  Diffusion models promise efficient parallel text generation but rely on bidirectional attention, creating a structural mismatch with pre-trained Autoregressive (AR) models.<n>We propose FLUID, a framework that efficiently adapts AR backbones to the diffusion paradigm.<n> Experiments demonstrate that FLUID achieves state-of-the-art performance while reducing training costs by orders of magnitude.
  arXiv  Detail & Related papers  (2026-04-11T13:18:27Z)
• Robustness Feature Adapter for Efficient Adversarial Training [6.88216456289286]
  Adversarial training (AT) with projected gradient descent is the most popular method to improve model robustness under adversarial attacks.<n> computational overheads become prohibitively large when AT is applied to large backbone models.<n>We propose a new adapter-based approach for efficient AT directly in the feature space.
  arXiv  Detail & Related papers  (2025-08-25T05:23:50Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization [35.922096876707975]
  PACE is a generalization of PArameter-efficient fine-tuning with Consistency rEgularization. It implicitly regularizes gradients for enhanced generalization, but also implicitly aligns the fine-tuned and pre-trained models to retain knowledge. It also improves LoRA in text classification (GLUE) and mathematical reasoning.
  arXiv  Detail & Related papers  (2024-09-25T17:56:00Z)
• Fourier-basis Functions to Bridge Augmentation Gap: Rethinking Frequency Augmentation in Image Classification [3.129187821625805]
  Auxiliary Fourier-basis Augmentation (AFA) is a technique targeting augmentation in the frequency domain and filling the augmentation gap left by visual augmentations. Our results show that AFA benefits the robustness of models against common corruptions, OOD generalization, and consistency of performance of models against increasing perturbations, with negligible deficit to the standard performance of models.
  arXiv  Detail & Related papers  (2024-03-04T11:30:02Z)
• Inference Stage Denoising for Undersampled MRI Reconstruction [13.8086726938161]
  Reconstruction of magnetic resonance imaging (MRI) data has been positively affected by deep learning. A key challenge remains: to improve generalisation to distribution shifts between the training and testing data.
  arXiv  Detail & Related papers  (2024-02-12T12:50:10Z)
• DASA: Difficulty-Aware Semantic Augmentation for Speaker Verification [55.306583814017046]
  We present a novel difficulty-aware semantic augmentation (DASA) approach for speaker verification. DASA generates diversified training samples in speaker embedding space with negligible extra computing cost. The best result achieves a 14.6% relative reduction in EER metric on CN-Celeb evaluation set.
  arXiv  Detail & Related papers  (2023-10-18T17:07:05Z)
• Data Augmentation Alone Can Improve Adversarial Training [5.1024659285813785]
  We find that data augmentation alone can significantly boost accuracy and robustness in adversarial training. We propose a new crop transformation, Cropshift, with much improved diversity and well-balanced hardness. Our augmentation method achieves the state-of-the-art accuracy and robustness for data augmentations in adversarial training.
  arXiv  Detail & Related papers  (2023-01-24T09:36:39Z)
• Efficient Few-Shot Object Detection via Knowledge Inheritance [62.36414544915032]
  Few-shot object detection (FSOD) aims at learning a generic detector that can adapt to unseen tasks with scarce training samples. We present an efficient pretrain-transfer framework (PTF) baseline with no computational increment. We also propose an adaptive length re-scaling (ALR) strategy to alleviate the vector length inconsistency between the predicted novel weights and the pretrained base weights.
  arXiv  Detail & Related papers  (2022-03-23T06:24:31Z)
• Make Some Noise: Reliable and Efficient Single-Step Adversarial Training [81.2112613657771]
  We revisit the role of noise and clipping in single-step adversarial training. We find that using a stronger noise around the clean sample combined with not clipping is highly effective in avoiding CO for large perturbation radii.
  arXiv  Detail & Related papers  (2022-02-02T18:10:01Z)
• Consistency Regularization for Adversarial Robustness [88.65786118562005]
  Adversarial training is one of the most successful methods to obtain the adversarial robustness of deep neural networks. However, a significant generalization gap in the robustness obtained from AT has been problematic. In this paper, we investigate data augmentation techniques to address the issue.
  arXiv  Detail & Related papers  (2021-03-08T09:21:41Z)
• Understanding and Improving Fast Adversarial Training [27.666483899332643]
  A recent line of work focused on making adversarial training computationally efficient for deep learning models. We show that adding a random step to FGSM does not prevent catastrophic overfitting. We propose a new regularization method, GradAlign, that prevents catastrophic overfitting.
  arXiv  Detail & Related papers  (2020-07-06T10:16:43Z)
• AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights [53.8489656709356]
  Normalization techniques are a boon for modern deep learning. It is often overlooked, however, that the additional introduction of momentum results in a rapid reduction in effective step sizes for scale-invariant weights. In this paper, we verify that the widely-adopted combination of the two ingredients lead to the premature decay of effective step sizes and sub-optimal model performances.
  arXiv  Detail & Related papers  (2020-06-15T08:35:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.