Supervised Batch Normalization

• URL: http://arxiv.org/abs/2405.17027v1
• Date: Mon, 27 May 2024 10:30:21 GMT
• Title: Supervised Batch Normalization
• Authors: Bilal Faye, Mustapha Lebbah, Hanane Azzag,
• Abstract summary: Batch Normalization (BN) is a widely-used technique in neural networks. We propose Supervised Batch Normalization (SBN), a pioneering approach. We define contexts as modes, categorizing data with similar characteristics.
• Score: 0.08192907805418585
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Batch Normalization (BN), a widely-used technique in neural networks, enhances generalization and expedites training by normalizing each mini-batch to the same mean and variance. However, its effectiveness diminishes when confronted with diverse data distributions. To address this challenge, we propose Supervised Batch Normalization (SBN), a pioneering approach. We expand normalization beyond traditional single mean and variance parameters, enabling the identification of data modes prior to training. This ensures effective normalization for samples sharing common features. We define contexts as modes, categorizing data with similar characteristics. These contexts are explicitly defined, such as domains in domain adaptation or modalities in multimodal systems, or implicitly defined through clustering algorithms based on data similarity. We illustrate the superiority of our approach over BN and other commonly employed normalization techniques through various experiments on both single and multi-task datasets. Integrating SBN with Vision Transformer results in a remarkable \textit{15.13}\% accuracy enhancement on CIFAR-100. Additionally, in domain adaptation scenarios, employing AdaMatch demonstrates an impressive \textit{22.25}\% accuracy improvement on MNIST and SVHN compared to BN.

Related papers

• Adaptative Context Normalization: A Boost for Deep Learning in Image Processing [0.07499722271664146]
  Adaptative Context Normalization (ACN) is a novel supervised approach that introduces the concept of "context" ACN ensures speed, convergence, and superior performance compared to BN and MN.
  arXiv  Detail & Related papers  (2024-09-07T08:18:10Z)
• Enhancing Neural Network Representations with Prior Knowledge-Based Normalization [0.07499722271664146]
  We introduce a new approach to multi-mode normalization that leverages prior knowledge to improve neural network representations. Our methods demonstrate superior convergence and performance across tasks in image classification, domain adaptation, and image generation.
  arXiv  Detail & Related papers  (2024-03-25T14:17:38Z)
• Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Framework [55.22949690864962]
  Batch Normalization (BN) has become an essential technique in contemporary neural network design. We propose a two-stage unified framework called Unified Batch Normalization (UBN) UBN significantly enhances performance across different visual backbones and different vision tasks.
  arXiv  Detail & Related papers  (2023-11-27T16:41:31Z)
• RegBN: Batch Normalization of Multimodal Data with Regularization [5.293979881130494]
  This paper introduces a novel approach for the normalization of multimodal data, called RegBN. RegBN uses the Frobenius norm as a regularizer term to address the side effects of confounders and underlying dependencies among different data sources. We validate the effectiveness of RegBN on eight databases from five research areas.
  arXiv  Detail & Related papers  (2023-10-01T11:05:45Z)
• NormAUG: Normalization-guided Augmentation for Domain Generalization [60.159546669021346]
  We propose a simple yet effective method called NormAUG (Normalization-guided Augmentation) for deep learning. Our method introduces diverse information at the feature level and improves the generalization of the main path. In the test stage, we leverage an ensemble strategy to combine the predictions from the auxiliary path of our model, further boosting performance.
  arXiv  Detail & Related papers  (2023-07-25T13:35:45Z)
• Single Domain Generalization via Normalised Cross-correlation Based Convolutions [14.306250516592304]
  Single Domain Generalization aims to train robust models using data from a single source. We propose a novel operator called XCNorm that computes the normalized cross-correlation between weights and an input feature patch. We show that deep neural networks composed of this operator are robust to common semantic distribution shifts.
  arXiv  Detail & Related papers  (2023-07-12T04:15:36Z)
• Patch-aware Batch Normalization for Improving Cross-domain Robustness [55.06956781674986]
  Cross-domain tasks present a challenge in which the model's performance will degrade when the training set and the test set follow different distributions. We propose a novel method called patch-aware batch normalization (PBN) By exploiting the differences between local patches of an image, our proposed PBN can effectively enhance the robustness of the model's parameters.
  arXiv  Detail & Related papers  (2023-04-06T03:25:42Z)
• Context Normalization Layer with Applications [0.1499944454332829]
  This study proposes a new normalization technique, called context normalization, for image data. It adjusts the scaling of features based on the characteristics of each sample, which improves the model's convergence speed and performance. The effectiveness of context normalization is demonstrated on various datasets, and its performance is compared to other standard normalization techniques.
  arXiv  Detail & Related papers  (2023-03-14T06:38:17Z)
• Generalizable Person Re-Identification via Self-Supervised Batch Norm Test-Time Adaption [63.7424680360004]
  Batch Norm Test-time Adaption (BNTA) is a novel re-id framework that applies the self-supervised strategy to update BN parameters adaptively. BNTA explores the domain-aware information within unlabeled target data before inference, and accordingly modulates the feature distribution normalized by BN to adapt to the target domain.
  arXiv  Detail & Related papers  (2022-03-01T18:46:32Z)
• Squared $\ell_2$ Norm as Consistency Loss for Leveraging Augmented Data to Learn Robust and Invariant Representations [76.85274970052762]
  Regularizing distance between embeddings/representations of original samples and augmented counterparts is a popular technique for improving robustness of neural networks. In this paper, we explore these various regularization choices, seeking to provide a general understanding of how we should regularize the embeddings. We show that the generic approach we identified (squared $ell$ regularized augmentation) outperforms several recent methods, which are each specially designed for one task.
  arXiv  Detail & Related papers  (2020-11-25T22:40:09Z)
• Double Forward Propagation for Memorized Batch Normalization [68.34268180871416]
  Batch Normalization (BN) has been a standard component in designing deep neural networks (DNNs) We propose a memorized batch normalization (MBN) which considers multiple recent batches to obtain more accurate and robust statistics. Compared to related methods, the proposed MBN exhibits consistent behaviors in both training and inference.
  arXiv  Detail & Related papers  (2020-10-10T08:48:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.