AFN: Adaptive Fusion Normalization via an Encoder-Decoder Framework

• URL: http://arxiv.org/abs/2308.03321v4
• Date: Mon, 19 Feb 2024 02:19:41 GMT
• Title: AFN: Adaptive Fusion Normalization via an Encoder-Decoder Framework
• Authors: Zikai Zhou, Shuo Zhang, Ziruo Wang, Huanran Chen
• Abstract summary: We propose a new normalization function called Adaptive Fusion Normalization. Through experiments, we demonstrate AFN outperforms the previous normalization techniques in domain generalization and image classification tasks.
• Score: 6.293148047652131
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The success of deep learning is inseparable from normalization layers. Researchers have proposed various normalization functions, and each of them has both advantages and disadvantages. In response, efforts have been made to design a unified normalization function that combines all normalization procedures and mitigates their weaknesses. We also proposed a new normalization function called Adaptive Fusion Normalization. Through experiments, we demonstrate AFN outperforms the previous normalization techniques in domain generalization and image classification tasks.

Related papers

• Error Feedback under $(L_0,L_1)$-Smoothness: Normalization and Momentum [56.37522020675243]
  We provide the first proof of convergence for normalized error feedback algorithms across a wide range of machine learning problems. We show that due to their larger allowable stepsizes, our new normalized error feedback algorithms outperform their non-normalized counterparts on various tasks.
  arXiv  Detail & Related papers  (2024-10-22T10:19:27Z)
• Unsupervised Adaptive Normalization [0.07499722271664146]
  Unsupervised Adaptive Normalization (UAN) is an innovative algorithm that seamlessly integrates clustering for normalization with deep neural network learning. UAN outperforms the classical methods by adapting to the target task and is effective in classification, and domain adaptation.
  arXiv  Detail & Related papers  (2024-09-07T08:14:11Z)
• SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients [99.13839450032408]
  It is desired to design a universal framework for adaptive algorithms to solve general problems. In particular, our novel framework provides adaptive methods under non convergence support for setting.
  arXiv  Detail & Related papers  (2021-06-15T15:16:28Z)
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence [8.411385346896413]
  We find that layer normalization and instance normalization both induce the appearance of failure modes in the neural network's pre-activations. We introduce the technique " Proxy Normalization" that normalizes post-activations using a proxy distribution. When combined with layer normalization or group normalization, this batch-independent normalization emulates batch normalization's behavior and consistently matches or exceeds its performance.
  arXiv  Detail & Related papers  (2021-06-07T16:08:48Z)
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application [111.82265258916397]
  Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs) This paper reviews and comments on the past, present and future of normalization methods in the context of training.
  arXiv  Detail & Related papers  (2020-09-27T13:06:52Z)
• A Flexible Framework for Designing Trainable Priors with Adaptive Smoothing and Game Encoding [57.1077544780653]
  We introduce a general framework for designing and training neural network layers whose forward passes can be interpreted as solving non-smooth convex optimization problems. We focus on convex games, solved by local agents represented by the nodes of a graph and interacting through regularization functions. This approach is appealing for solving imaging problems, as it allows the use of classical image priors within deep models that are trainable end to end.
  arXiv  Detail & Related papers  (2020-06-26T08:34:54Z)
• Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction [3.201333208812837]
  We propose a new and effective normalization approaches based on LayerNorm named variance only LayerNorm(VO-LN) in this work. We find that the variance of normalization plays the main role and give an explanation in this work.
  arXiv  Detail & Related papers  (2020-06-23T04:35:22Z)
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers [82.61117235807606]
  The success of deep neural networks in part due to the use of normalization layers. Our analysis shows how the introduction of normalization changes the landscape and can enable faster activation.
  arXiv  Detail & Related papers  (2020-06-11T23:55:54Z)
• Stochastic batch size for adaptive regularization in deep network optimization [63.68104397173262]
  We propose a first-order optimization algorithm incorporating adaptive regularization applicable to machine learning problems in deep learning framework. We empirically demonstrate the effectiveness of our algorithm using an image classification task based on conventional network models applied to commonly used benchmark datasets.
  arXiv  Detail & Related papers  (2020-04-14T07:54:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.