AFN: Adaptive Fusion Normalization via an Encoder-Decoder Framework
- URL: http://arxiv.org/abs/2308.03321v4
- Date: Mon, 19 Feb 2024 02:19:41 GMT
- Title: AFN: Adaptive Fusion Normalization via an Encoder-Decoder Framework
- Authors: Zikai Zhou, Shuo Zhang, Ziruo Wang, Huanran Chen
- Abstract summary: We propose a new normalization function called Adaptive Fusion Normalization.
Through experiments, we demonstrate AFN outperforms the previous normalization techniques in domain generalization and image classification tasks.
- Score: 6.293148047652131
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The success of deep learning is inseparable from normalization layers.
Researchers have proposed various normalization functions, and each of them has
both advantages and disadvantages. In response, efforts have been made to
design a unified normalization function that combines all normalization
procedures and mitigates their weaknesses. We also proposed a new normalization
function called Adaptive Fusion Normalization. Through experiments, we
demonstrate AFN outperforms the previous normalization techniques in domain
generalization and image classification tasks.
Related papers
- Context Normalization Layer with Applications [0.1499944454332829]
This study proposes a new normalization technique, called context normalization, for image data.
It adjusts the scaling of features based on the characteristics of each sample, which improves the model's convergence speed and performance.
The effectiveness of context normalization is demonstrated on various datasets, and its performance is compared to other standard normalization techniques.
arXiv Detail & Related papers (2023-03-14T06:38:17Z) - SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients [99.13839450032408]
It is desired to design a universal framework for adaptive algorithms to solve general problems.
In particular, our novel framework provides adaptive methods under non convergence support for setting.
arXiv Detail & Related papers (2021-06-15T15:16:28Z) - Proxy-Normalizing Activations to Match Batch Normalization while
Removing Batch Dependence [8.411385346896413]
We find that layer normalization and instance normalization both induce the appearance of failure modes in the neural network's pre-activations.
We introduce the technique " Proxy Normalization" that normalizes post-activations using a proxy distribution.
When combined with layer normalization or group normalization, this batch-independent normalization emulates batch normalization's behavior and consistently matches or exceeds its performance.
arXiv Detail & Related papers (2021-06-07T16:08:48Z) - Rethinking Skip Connection with Layer Normalization in Transformers and
ResNets [49.87919454950763]
Skip connection is a widely-used technique to improve the performance of deep neural networks.
In this work, we investigate how the scale factors in the effectiveness of the skip connection.
arXiv Detail & Related papers (2021-05-15T11:44:49Z) - Normalization Techniques in Training DNNs: Methodology, Analysis and
Application [111.82265258916397]
Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs)
This paper reviews and comments on the past, present and future of normalization methods in the context of training.
arXiv Detail & Related papers (2020-09-27T13:06:52Z) - A Flexible Framework for Designing Trainable Priors with Adaptive
Smoothing and Game Encoding [57.1077544780653]
We introduce a general framework for designing and training neural network layers whose forward passes can be interpreted as solving non-smooth convex optimization problems.
We focus on convex games, solved by local agents represented by the nodes of a graph and interacting through regularization functions.
This approach is appealing for solving imaging problems, as it allows the use of classical image priors within deep models that are trainable end to end.
arXiv Detail & Related papers (2020-06-26T08:34:54Z) - Correct Normalization Matters: Understanding the Effect of Normalization
On Deep Neural Network Models For Click-Through Rate Prediction [3.201333208812837]
We propose a new and effective normalization approaches based on LayerNorm named variance only LayerNorm(VO-LN) in this work.
We find that the variance of normalization plays the main role and give an explanation in this work.
arXiv Detail & Related papers (2020-06-23T04:35:22Z) - Optimization Theory for ReLU Neural Networks Trained with Normalization
Layers [82.61117235807606]
The success of deep neural networks in part due to the use of normalization layers.
Our analysis shows how the introduction of normalization changes the landscape and can enable faster activation.
arXiv Detail & Related papers (2020-06-11T23:55:54Z) - Stochastic batch size for adaptive regularization in deep network
optimization [63.68104397173262]
We propose a first-order optimization algorithm incorporating adaptive regularization applicable to machine learning problems in deep learning framework.
We empirically demonstrate the effectiveness of our algorithm using an image classification task based on conventional network models applied to commonly used benchmark datasets.
arXiv Detail & Related papers (2020-04-14T07:54:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.