Enhancing Neural Network Representations with Prior Knowledge-Based Normalization

• URL: http://arxiv.org/abs/2403.16798v3
• Date: Wed, 30 Oct 2024 10:55:01 GMT
• Title: Enhancing Neural Network Representations with Prior Knowledge-Based Normalization
• Authors: Bilal Faye, Hanane Azzag, Mustapha Lebbah, Djamel Bouchaffra,
• Abstract summary: We introduce a new approach to multi-mode normalization that leverages prior knowledge to improve neural network representations. Our methods demonstrate superior convergence and performance across tasks in image classification, domain adaptation, and image generation.
• Score: 0.07499722271664146
• License:
• Abstract: Deep learning models face persistent challenges in training, particularly due to internal covariate shift and label shift. While single-mode normalization methods like Batch Normalization partially address these issues, they are constrained by batch size dependencies and limiting distributional assumptions. Multi-mode normalization techniques mitigate these limitations but struggle with computational demands when handling diverse Gaussian distributions. In this paper, we introduce a new approach to multi-mode normalization that leverages prior knowledge to improve neural network representations. Our method organizes data into predefined structures, or "contexts", prior to training and normalizes based on these contexts, with two variants: Context Normalization (CN) and Context Normalization - Extended (CN-X). When contexts are unavailable, we introduce Adaptive Context Normalization (ACN), which dynamically builds contexts in the latent space during training. Across tasks in image classification, domain adaptation, and image generation, our methods demonstrate superior convergence and performance.

Related papers

• Adaptative Context Normalization: A Boost for Deep Learning in Image Processing [0.07499722271664146]
  Adaptative Context Normalization (ACN) is a novel supervised approach that introduces the concept of "context" ACN ensures speed, convergence, and superior performance compared to BN and MN.
  arXiv  Detail & Related papers  (2024-09-07T08:18:10Z)
• Unsupervised Adaptive Normalization [0.07499722271664146]
  Unsupervised Adaptive Normalization (UAN) is an innovative algorithm that seamlessly integrates clustering for normalization with deep neural network learning. UAN outperforms the classical methods by adapting to the target task and is effective in classification, and domain adaptation.
  arXiv  Detail & Related papers  (2024-09-07T08:14:11Z)
• NormAUG: Normalization-guided Augmentation for Domain Generalization [60.159546669021346]
  We propose a simple yet effective method called NormAUG (Normalization-guided Augmentation) for deep learning. Our method introduces diverse information at the feature level and improves the generalization of the main path. In the test stage, we leverage an ensemble strategy to combine the predictions from the auxiliary path of our model, further boosting performance.
  arXiv  Detail & Related papers  (2023-07-25T13:35:45Z)
• Neural Fields with Hard Constraints of Arbitrary Differential Order [61.49418682745144]
  We develop a series of approaches for enforcing hard constraints on neural fields. The constraints can be specified as a linear operator applied to the neural field and its derivatives. Our approaches are demonstrated in a wide range of real-world applications.
  arXiv  Detail & Related papers  (2023-06-15T08:33:52Z)
• Context Normalization Layer with Applications [0.1499944454332829]
  This study proposes a new normalization technique, called context normalization, for image data. It adjusts the scaling of features based on the characteristics of each sample, which improves the model's convergence speed and performance. The effectiveness of context normalization is demonstrated on various datasets, and its performance is compared to other standard normalization techniques.
  arXiv  Detail & Related papers  (2023-03-14T06:38:17Z)
• Breaking Time Invariance: Assorted-Time Normalization for RNNs [5.229616140749998]
  We propose a normalization method called Assorted-Time Normalization (ATN) ATN preserves information from multiple consecutive time steps and normalizes using them. Our experiments applying ATN to LN demonstrate consistent improvement on various tasks.
  arXiv  Detail & Related papers  (2022-09-28T21:51:13Z)
• Subquadratic Overparameterization for Shallow Neural Networks [60.721751363271146]
  We provide an analytical framework that allows us to adopt standard neural training strategies. We achieve the desiderata viaak-Lojasiewicz, smoothness, and standard assumptions.
  arXiv  Detail & Related papers  (2021-11-02T20:24:01Z)
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application [111.82265258916397]
  Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs) This paper reviews and comments on the past, present and future of normalization methods in the context of training.
  arXiv  Detail & Related papers  (2020-09-27T13:06:52Z)
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers [82.61117235807606]
  The success of deep neural networks in part due to the use of normalization layers. Our analysis shows how the introduction of normalization changes the landscape and can enable faster activation.
  arXiv  Detail & Related papers  (2020-06-11T23:55:54Z)
• AdaS: Adaptive Scheduling of Stochastic Gradients [50.80697760166045]
  We introduce the notions of textit"knowledge gain" and textit"mapping condition" and propose a new algorithm called Adaptive Scheduling (AdaS) Experimentation reveals that, using the derived metrics, AdaS exhibits: (a) faster convergence and superior generalization over existing adaptive learning methods; and (b) lack of dependence on a validation set to determine when to stop training.
  arXiv  Detail & Related papers  (2020-06-11T16:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.