Towards an Adversarially Robust Normalization Approach

• URL: http://arxiv.org/abs/2006.11007v1
• Date: Fri, 19 Jun 2020 08:12:25 GMT
• Title: Towards an Adversarially Robust Normalization Approach
• Authors: Muhammad Awais, Fahad Shamshad, Sung-Ho Bae
• Abstract summary: Batch Normalization (BatchNorm) is effective for improving the performance and accelerating the training of deep neural networks. It has also shown to be a cause of adversarial vulnerability, i.e., networks without it are more robust to adversarial attacks. We propose Robust Normalization (RobustNorm); an adversarially robust version of BatchNorm.
• Score: 8.744644782067368
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Batch Normalization (BatchNorm) is effective for improving the performance and accelerating the training of deep neural networks. However, it has also shown to be a cause of adversarial vulnerability, i.e., networks without it are more robust to adversarial attacks. In this paper, we investigate how BatchNorm causes this vulnerability and proposed new normalization that is robust to adversarial attacks. We first observe that adversarial images tend to shift the distribution of BatchNorm input, and this shift makes train-time estimated population statistics inaccurate. We hypothesize that these inaccurate statistics make models with BatchNorm more vulnerable to adversarial attacks. We prove our hypothesis by replacing train-time estimated statistics with statistics calculated from the inference-time batch. We found that the adversarial vulnerability of BatchNorm disappears if we use these statistics. However, without estimated batch statistics, we can not use BatchNorm in the practice if large batches of input are not available. To mitigate this, we propose Robust Normalization (RobustNorm); an adversarially robust version of BatchNorm. We experimentally show that models trained with RobustNorm perform better in adversarial settings while retaining all the benefits of BatchNorm. Code is available at \url{https://github.com/awaisrauf/RobustNorm}.

Related papers

• Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Off [107.35833747750446]
  adversarial examples can be crafted by adding imperceptible perturbations to legitimate documents. This vulnerability raises significant concerns about their reliability and hinders the widespread deployment of NRMs. In this study, we establish theoretical guarantees regarding the effectiveness-robustness trade-off in NRMs.
  arXiv  Detail & Related papers  (2023-12-16T05:38:39Z)
• An Empirical Analysis of the Shift and Scale Parameters in BatchNorm [3.198144010381572]
  Batch Normalization (BatchNorm) is a technique that improves the training of deep neural networks. This paper examines the relative contribution to the success of BatchNorm of the normalization step.
  arXiv  Detail & Related papers  (2023-03-22T12:41:12Z)
• Removing Batch Normalization Boosts Adversarial Training [83.08844497295148]
  Adversarial training (AT) defends deep neural networks against adversarial attacks. A major bottleneck is the widely used batch normalization (BN), which struggles to model the different statistics of clean and adversarial training samples in AT. Our normalizer-free robust training (NoFrost) method extends recent advances in normalizer-free networks to AT.
  arXiv  Detail & Related papers  (2022-07-04T01:39:37Z)
• Distributed Adversarial Training to Robustify Deep Neural Networks at Scale [100.19539096465101]
  Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification. To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training. We propose a large-batch adversarial training framework implemented over multiple machines.
  arXiv  Detail & Related papers  (2022-06-13T15:39:43Z)
• Test-time Batch Statistics Calibration for Covariate Shift [66.7044675981449]
  We propose to adapt the deep models to the novel environment during inference. We present a general formulation $alpha$-BN to calibrate the batch statistics. We also present a novel loss function to form a unified test time adaptation framework Core.
  arXiv  Detail & Related papers  (2021-10-06T08:45:03Z)
• Rethinking "Batch" in BatchNorm [25.69755850518617]
  BatchNorm is a critical building block in modern convolutional neural networks. This paper thoroughly reviews such problems in visual recognition tasks, and shows that a key to address them is to rethink different choices in the concept of "batch" in BatchNorm.
  arXiv  Detail & Related papers  (2021-05-17T01:58:15Z)
• Towards Defending Multiple $\ell_p$-norm Bounded Adversarial Perturbations via Gated Batch Normalization [120.99395850108422]
  Existing adversarial defenses typically improve model robustness against individual specific perturbations. Some recent methods improve model robustness against adversarial attacks in multiple $ell_p$ balls, but their performance against each perturbation type is still far from satisfactory. We propose Gated Batch Normalization (GBN) to adversarially train a perturbation-invariant predictor for defending multiple $ell_p bounded adversarial perturbations.
  arXiv  Detail & Related papers  (2020-12-03T02:26:01Z)
• Does Data Augmentation Benefit from Split BatchNorms [29.134017115737507]
  State-of-the-art data augmentation strongly distorts training images, leading to a disparity between examples seen during training and inference. We propose an auxiliary BatchNorm for the potentially out-of-distribution, strongly augmented images. We find that this method significantly improves the performance of common image classification benchmarks such as CIFAR-10, CIFAR-100, and ImageNet.
  arXiv  Detail & Related papers  (2020-10-15T15:00:43Z)
• A New Look at Ghost Normalization [12.331754048486554]
  Ghost normalization (GhostNorm) has been shown to improve upon BatchNorm in some datasets. Our contributions are: (i) we uncover a source of regularization that is unique to GhostNorm, and not simply an extension from BatchNorm, and (ii) three types of GhostNorm implementations are described.
  arXiv  Detail & Related papers  (2020-07-16T18:23:52Z)
• Separating the Effects of Batch Normalization on CNN Training Speed and Stability Using Classical Adaptive Filter Theory [40.55789598448379]
  Batch Normalization (BatchNorm) is commonly used in Convolutional Neural Networks (CNNs) to improve training speed and stability. This paper uses concepts from the traditional adaptive filter domain to provide insight into the dynamics and inner workings of BatchNorm.
  arXiv  Detail & Related papers  (2020-02-25T05:25:40Z)
• Cross-Iteration Batch Normalization [67.83430009388678]
  We present Cross-It Batch Normalization (CBN), in which examples from multiple recent iterations are jointly utilized to enhance estimation quality. CBN is found to outperform the original batch normalization and a direct calculation of statistics over previous iterations without the proposed compensation technique.
  arXiv  Detail & Related papers  (2020-02-13T18:52:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.