Exemplar Normalization for Learning Deep Representation

• URL: http://arxiv.org/abs/2003.08761v2
• Date: Fri, 20 Mar 2020 14:58:40 GMT
• Title: Exemplar Normalization for Learning Deep Representation
• Authors: Ruimao Zhang, Zhanglin Peng, Lingyun Wu, Zhen Li, Ping Luo
• Abstract summary: This work investigates a novel dynamic learning-to-normalize (L2N) problem by proposing Exemplar Normalization (EN) EN is able to learn different normalization methods for different convolutional layers and image samples of a deep network.
• Score: 34.42934843556172
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Normalization techniques are important in different advanced neural networks and different tasks. This work investigates a novel dynamic learning-to-normalize (L2N) problem by proposing Exemplar Normalization (EN), which is able to learn different normalization methods for different convolutional layers and image samples of a deep network. EN significantly improves flexibility of the recently proposed switchable normalization (SN), which solves a static L2N problem by linearly combining several normalizers in each normalization layer (the combination is the same for all samples). Instead of directly employing a multi-layer perceptron (MLP) to learn data-dependent parameters as conditional batch normalization (cBN) did, the internal architecture of EN is carefully designed to stabilize its optimization, leading to many appealing benefits. (1) EN enables different convolutional layers, image samples, categories, benchmarks, and tasks to use different normalization methods, shedding light on analyzing them in a holistic view. (2) EN is effective for various network architectures and tasks. (3) It could replace any normalization layers in a deep network and still produce stable model training. Extensive experiments demonstrate the effectiveness of EN in a wide spectrum of tasks including image recognition, noisy label learning, and semantic segmentation. For example, by replacing BN in the ordinary ResNet50, improvement produced by EN is 300% more than that of SN on both ImageNet and the noisy WebVision dataset.

Related papers

• Multi-Objective Optimization for Sparse Deep Multi-Task Learning [0.0]
  We present a Multi-Objective Optimization algorithm using a modified Weighted Chebyshev scalarization for training Deep Neural Networks (DNNs) Our work aims to address the (economical and also ecological) sustainability issue of DNN models, with particular focus on Deep Multi-Task models.
  arXiv  Detail & Related papers  (2023-08-23T16:42:27Z)
• WLD-Reg: A Data-dependent Within-layer Diversity Regularizer [98.78384185493624]
  Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization. We propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. We present an extensive empirical study confirming that the proposed approach enhances the performance of several state-of-the-art neural network models in multiple tasks.
  arXiv  Detail & Related papers  (2023-01-03T20:57:22Z)
• Subquadratic Overparameterization for Shallow Neural Networks [60.721751363271146]
  We provide an analytical framework that allows us to adopt standard neural training strategies. We achieve the desiderata viaak-Lojasiewicz, smoothness, and standard assumptions.
  arXiv  Detail & Related papers  (2021-11-02T20:24:01Z)
• Exploiting Invariance in Training Deep Neural Networks [4.169130102668252]
  Inspired by two basic mechanisms in animal visual systems, we introduce a feature transform technique that imposes invariance properties in the training of deep neural networks. The resulting algorithm requires less parameter tuning, trains well with an initial learning rate 1.0, and easily generalizes to different tasks. Tested on ImageNet, MS COCO, and Cityscapes datasets, our proposed technique requires fewer iterations to train, surpasses all baselines by a large margin, seamlessly works on both small and large batch size training, and applies to different computer vision tasks of image classification, object detection, and semantic segmentation.
  arXiv  Detail & Related papers  (2021-03-30T19:18:31Z)
• Style Normalization and Restitution for DomainGeneralization and Adaptation [88.86865069583149]
  An effective domain generalizable model is expected to learn feature representations that are both generalizable and discriminative. In this paper, we design a novel Style Normalization and Restitution module (SNR) to ensure both high generalization and discrimination capability of the networks.
  arXiv  Detail & Related papers  (2021-01-03T09:01:39Z)
• Smoother Network Tuning and Interpolation for Continuous-level Image Processing [7.730087303035803]
  Filter Transition Network (FTN) is a structurally smoother module for continuous-level learning. FTN generalizes well across various tasks and networks and cause fewer undesirable side effects. For stable learning of FTN, we additionally propose a method to non-linear neural network layers with identity mappings.
  arXiv  Detail & Related papers  (2020-10-05T18:29:52Z)
• A Flexible Framework for Designing Trainable Priors with Adaptive Smoothing and Game Encoding [57.1077544780653]
  We introduce a general framework for designing and training neural network layers whose forward passes can be interpreted as solving non-smooth convex optimization problems. We focus on convex games, solved by local agents represented by the nodes of a graph and interacting through regularization functions. This approach is appealing for solving imaging problems, as it allows the use of classical image priors within deep models that are trainable end to end.
  arXiv  Detail & Related papers  (2020-06-26T08:34:54Z)
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers [82.61117235807606]
  The success of deep neural networks in part due to the use of normalization layers. Our analysis shows how the introduction of normalization changes the landscape and can enable faster activation.
  arXiv  Detail & Related papers  (2020-06-11T23:55:54Z)
• Evolving Normalization-Activation Layers [100.82879448303805]
  We develop efficient rejection protocols to quickly filter out candidate layers that do not work well. Our method leads to the discovery of EvoNorms, a set of new normalization-activation layers with novel, and sometimes surprising structures. Our experiments show that EvoNorms work well on image classification models including ResNets, MobileNets and EfficientNets.
  arXiv  Detail & Related papers  (2020-04-06T19:52:48Z)
• Regularized Adaptation for Stable and Efficient Continuous-Level Learning on Image Processing Networks [7.730087303035803]
  We propose a novel continuous-level learning framework using a Filter Transition Network (FTN) FTN is a non-linear module that easily adapt to new levels, and is regularized to prevent undesirable side-effects. Extensive results for various image processing indicate that the performance of FTN is stable in terms of adaptation and adaptation.
  arXiv  Detail & Related papers  (2020-03-11T07:46:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.