An Adaptive Batch Normalization in Deep Learning

• URL: http://arxiv.org/abs/2211.02050v1
• Date: Thu, 3 Nov 2022 12:12:56 GMT
• Title: An Adaptive Batch Normalization in Deep Learning
• Authors: Wael Alsobhi, Tarik Alafif, Alaa Abdel-Hakim, Weiwei Zong
• Abstract summary: Batch Normalization (BN) is a way to accelerate and stabilize training in deep convolutional neural networks. We propose a threshold-based adaptive BN approach that separates the data that requires the BN and data that does not require it.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Batch Normalization (BN) is a way to accelerate and stabilize training in deep convolutional neural networks. However, the BN works continuously within the network structure, although some training data may not always require it. In this research work, we propose a threshold-based adaptive BN approach that separates the data that requires the BN and data that does not require it. The experimental evaluation demonstrates that proposed approach achieves better performance mostly in small batch sizes than the traditional BN using MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100. It also reduces the occurrence of internal variable transformation to increase network stability

Related papers

• Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Framework [55.22949690864962]
  Batch Normalization (BN) has become an essential technique in contemporary neural network design. We propose a two-stage unified framework called Unified Batch Normalization (UBN) UBN significantly enhances performance across different visual backbones and different vision tasks.
  arXiv  Detail & Related papers  (2023-11-27T16:41:31Z)
• Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation [67.77048565738728]
  Continual learning involves learning a sequence of tasks and balancing their knowledge appropriately. We propose Adaptive Balance of BN (AdaB$2$N), which incorporates appropriately a Bayesian-based strategy to adapt task-wise contributions. Our approach achieves significant performance gains across a wide range of benchmarks.
  arXiv  Detail & Related papers  (2023-10-13T04:50:40Z)
• Rebalancing Batch Normalization for Exemplar-based Class-Incremental Learning [23.621259845287824]
  Batch Normalization (BN) has been extensively studied for neural nets in various computer vision tasks. We develop a new update patch for BN, particularly tailored for the exemplar-based class-incremental learning (CIL)
  arXiv  Detail & Related papers  (2022-01-29T11:03:03Z)
• Revisiting Batch Normalization [0.0]
  Batch normalization (BN) is essential for training deep neural networks. We revisit the BN formulation and present a new method and update approach for BN to address the aforementioned issues. Experimental results using the proposed alterations to BN show statistically significant performance gains in a variety of scenarios. We also present a new online BN-based input data normalization technique to alleviate the need for other offline or fixed methods.
  arXiv  Detail & Related papers  (2021-10-26T19:48:19Z)
• Batch Normalization Preconditioning for Neural Network Training [7.709342743709842]
  Batch normalization (BN) is a popular and ubiquitous method in deep learning. BN is not suitable for use with very small mini-batch sizes or online learning. We propose a new method called Batch Normalization Preconditioning (BNP)
  arXiv  Detail & Related papers  (2021-08-02T18:17:26Z)
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization [92.23297927690149]
  Batch normalization (BN) is a key facilitator and considered essential for state-of-the-art binary neural networks (BNN) We extend their framework to training BNNs, and for the first time demonstrate that BNs can be completed removed from BNN training and inference regimes.
  arXiv  Detail & Related papers  (2021-04-16T16:46:57Z)
• MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization [60.36100335878855]
  We propose a novel normalization method, named MimicNorm, to improve the convergence and efficiency in network training. We leverage the neural kernel (NTK) theory to prove that our weight mean operation whitens activations and transits network into the chaotic regime like BN layer. MimicNorm achieves similar accuracy for various network structures, including ResNets and lightweight networks like ShuffleNet, with a reduction of about 20% memory consumption.
  arXiv  Detail & Related papers  (2020-10-19T07:42:41Z)
• Double Forward Propagation for Memorized Batch Normalization [68.34268180871416]
  Batch Normalization (BN) has been a standard component in designing deep neural networks (DNNs) We propose a memorized batch normalization (MBN) which considers multiple recent batches to obtain more accurate and robust statistics. Compared to related methods, the proposed MBN exhibits consistent behaviors in both training and inference.
  arXiv  Detail & Related papers  (2020-10-10T08:48:41Z)
• Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization [126.6252371899064]
  Moving Average Batch Normalization (MABN) is a novel normalization method. We show that MABN can completely restore the performance of vanilla BN in small batch cases. Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO.
  arXiv  Detail & Related papers  (2020-01-19T14:41:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.