Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression

• URL: http://arxiv.org/abs/2102.02950v1
• Date: Fri, 5 Feb 2021 01:31:01 GMT
• Title: Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression
• Authors: Masanori Yamada, Sekitoshi Kanai, Tomoharu Iwata, Tomokatsu Takahashi, Yuki Yamanaka, Hiroshi Takahashi, Atsutoshi Kumagai
• Abstract summary: Adrial training is actively studied for learning robust models against adversarial examples. A recent study finds that adversarially trained models degenerate performance on adversarial examples when their weight loss landscape is sharp. We theoretically analyze this phenomenon in this paper.
• Score: 45.34758512755516
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adversarial training is actively studied for learning robust models against adversarial examples. A recent study finds that adversarially trained models degenerate generalization performance on adversarial examples when their weight loss landscape, which is loss changes with respect to weights, is sharp. Unfortunately, it has been experimentally shown that adversarial training sharpens the weight loss landscape, but this phenomenon has not been theoretically clarified. Therefore, we theoretically analyze this phenomenon in this paper. As a first step, this paper proves that adversarial training with the L2 norm constraints sharpens the weight loss landscape in the linear logistic regression model. Our analysis reveals that the sharpness of the weight loss landscape is caused by the noise aligned in the direction of increasing the loss, which is used in adversarial training. We theoretically and experimentally confirm that the weight loss landscape becomes sharper as the magnitude of the noise of adversarial training increases in the linear logistic regression model. Moreover, we experimentally confirm the same phenomena in ResNet18 with softmax as a more general case.

Related papers

• The Epochal Sawtooth Effect: Unveiling Training Loss Oscillations in Adam and Other Optimizers [8.770864706004472]
  We identify and analyze a recurring training loss pattern, which we term the textitEpochal Sawtooth Effect (ESE) This pattern is characterized by a sharp drop in loss at the beginning of each epoch, followed by a gradual increase, resulting in a sawtooth-shaped loss curve. We provide an in-depth explanation of the underlying mechanisms that lead to the Epochal Sawtooth Effect.
  arXiv  Detail & Related papers  (2024-10-14T00:51:21Z)
• Improving Fast Adversarial Training Paradigm: An Example Taxonomy Perspective [61.38753850236804]
  Fast adversarial training (FAT) is presented for efficient training and has become a hot research topic. FAT suffers from catastrophic overfitting, which leads to a performance drop compared with multi-step adversarial training. We present an example taxonomy in FAT, which identifies that catastrophic overfitting is caused by the imbalance between the inner and outer optimization in FAT.
  arXiv  Detail & Related papers  (2024-07-22T03:56:27Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Small-scale proxies for large-scale Transformer training instabilities [69.36381318171338]
  We seek ways to reproduce and study training stability and instability at smaller scales. By measuring the relationship between learning rate and loss across scales, we show that these instabilities also appear in small models when training at high learning rates. We study methods such as warm-up, weight decay, and the $mu$Param to train small models that achieve similar losses across orders of magnitude of learning rate variation.
  arXiv  Detail & Related papers  (2023-09-25T17:48:51Z)
• Understanding and Combating Robust Overfitting via Input Loss Landscape Analysis and Regularization [5.1024659285813785]
  Adrial training is prone to overfitting, and the cause is far from clear. We find that robust overfitting results from standard training, specifically the minimization of the clean loss. We propose a new regularizer to smooth the loss landscape by penalizing the weighted logits variation along the adversarial direction.
  arXiv  Detail & Related papers  (2022-12-09T16:55:30Z)
• Adversarial Weight Perturbation Helps Robust Generalization [65.68598525492666]
  Adversarial training is the most promising way to improve the robustness of deep neural networks against adversarial examples. We show how the widely used weight loss landscape (loss change with respect to weight) performs in adversarial training. We propose a simple yet effective Adversarial Weight Perturbation (AWP) to explicitly regularize the flatness of weight loss landscape.
  arXiv  Detail & Related papers  (2020-04-13T12:05:01Z)
• Overfitting in adversarially robust deep learning [86.11788847990783]
  We show that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training. We also show that effects such as the double descent curve do still occur in adversarially trained models, yet fail to explain the observed overfitting.
  arXiv  Detail & Related papers  (2020-02-26T15:40:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.