Revisiting Batch Normalization

• URL: http://arxiv.org/abs/2110.13989v1
• Date: Tue, 26 Oct 2021 19:48:19 GMT
• Title: Revisiting Batch Normalization
• Authors: Jim Davis and Logan Frank
• Abstract summary: Batch normalization (BN) is essential for training deep neural networks. We revisit the BN formulation and present a new method and update approach for BN to address the aforementioned issues. Experimental results using the proposed alterations to BN show statistically significant performance gains in a variety of scenarios. We also present a new online BN-based input data normalization technique to alleviate the need for other offline or fixed methods.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Batch normalization (BN) is comprised of a normalization component followed by an affine transformation and has become essential for training deep neural networks. Standard initialization of each BN in a network sets the affine transformation scale and shift to 1 and 0, respectively. However, after training we have observed that these parameters do not alter much from their initialization. Furthermore, we have noticed that the normalization process can still yield overly large values, which is undesirable for training. We revisit the BN formulation and present a new initialization method and update approach for BN to address the aforementioned issues. Experimental results using the proposed alterations to BN show statistically significant performance gains in a variety of scenarios. The approach can be used with existing implementations at no additional computational cost. We also present a new online BN-based input data normalization technique to alleviate the need for other offline or fixed methods. Source code is available at https://github.com/osu-cvl/revisiting-bn.

Related papers

• Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Framework [55.22949690864962]
  Batch Normalization (BN) has become an essential technique in contemporary neural network design. We propose a two-stage unified framework called Unified Batch Normalization (UBN) UBN significantly enhances performance across different visual backbones and different vision tasks.
  arXiv  Detail & Related papers  (2023-11-27T16:41:31Z)
• An Adaptive Batch Normalization in Deep Learning [0.0]
  Batch Normalization (BN) is a way to accelerate and stabilize training in deep convolutional neural networks. We propose a threshold-based adaptive BN approach that separates the data that requires the BN and data that does not require it.
  arXiv  Detail & Related papers  (2022-11-03T12:12:56Z)
• Rebalancing Batch Normalization for Exemplar-based Class-Incremental Learning [23.621259845287824]
  Batch Normalization (BN) has been extensively studied for neural nets in various computer vision tasks. We develop a new update patch for BN, particularly tailored for the exemplar-based class-incremental learning (CIL)
  arXiv  Detail & Related papers  (2022-01-29T11:03:03Z)
• Batch Normalization Preconditioning for Neural Network Training [7.709342743709842]
  Batch normalization (BN) is a popular and ubiquitous method in deep learning. BN is not suitable for use with very small mini-batch sizes or online learning. We propose a new method called Batch Normalization Preconditioning (BNP)
  arXiv  Detail & Related papers  (2021-08-02T18:17:26Z)
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization [92.23297927690149]
  Batch normalization (BN) is a key facilitator and considered essential for state-of-the-art binary neural networks (BNN) We extend their framework to training BNNs, and for the first time demonstrate that BNs can be completed removed from BNN training and inference regimes.
  arXiv  Detail & Related papers  (2021-04-16T16:46:57Z)
• MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization [60.36100335878855]
  We propose a novel normalization method, named MimicNorm, to improve the convergence and efficiency in network training. We leverage the neural kernel (NTK) theory to prove that our weight mean operation whitens activations and transits network into the chaotic regime like BN layer. MimicNorm achieves similar accuracy for various network structures, including ResNets and lightweight networks like ShuffleNet, with a reduction of about 20% memory consumption.
  arXiv  Detail & Related papers  (2020-10-19T07:42:41Z)
• Double Forward Propagation for Memorized Batch Normalization [68.34268180871416]
  Batch Normalization (BN) has been a standard component in designing deep neural networks (DNNs) We propose a memorized batch normalization (MBN) which considers multiple recent batches to obtain more accurate and robust statistics. Compared to related methods, the proposed MBN exhibits consistent behaviors in both training and inference.
  arXiv  Detail & Related papers  (2020-10-10T08:48:41Z)
• PowerNorm: Rethinking Batch Normalization in Transformers [96.14956636022957]
  normalization method for neural network (NN) models used in Natural Language Processing (NLP) is layer normalization (LN) LN is preferred due to the empirical observation that a (naive/vanilla) use of BN leads to significant performance degradation for NLP tasks. We propose Power Normalization (PN), a novel normalization scheme that resolves this issue.
  arXiv  Detail & Related papers  (2020-03-17T17:50:26Z)
• Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization [126.6252371899064]
  Moving Average Batch Normalization (MABN) is a novel normalization method. We show that MABN can completely restore the performance of vanilla BN in small batch cases. Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO.
  arXiv  Detail & Related papers  (2020-01-19T14:41:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.