Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-wise Relevance Propagation

• URL: http://arxiv.org/abs/2002.11018v1
• Date: Mon, 24 Feb 2020 13:06:55 GMT
• Title: Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-wise Relevance Propagation
• Authors: Mathilde Guillemot, Catherine Heusele, Rodolphe Korichi, Sylvianne Schnebert, Liming Chen
• Abstract summary: We build an equivalent network fusing normalization layers and convolutional or fully connected layers. Heatmaps obtained with our method on MNIST and CIFAR 10 datasets are more accurate for convolutional layers.
• Score: 2.654526698055524
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The lack of transparency of neural networks stays a major break for their use. The Layerwise Relevance Propagation technique builds heat-maps representing the relevance of each input in the model s decision. The relevance spreads backward from the last to the first layer of the Deep Neural Network. Layer-wise Relevance Propagation does not manage normalization layers, in this work we suggest a method to include normalization layers. Specifically, we build an equivalent network fusing normalization layers and convolutional or fully connected layers. Heatmaps obtained with our method on MNIST and CIFAR 10 datasets are more accurate for convolutional layers. Our study also prevents from using Layerwise Relevance Propagation with networks including a combination of connected layers and normalization layer.

Related papers

• Sensitivity-Based Layer Insertion for Residual and Feedforward Neural Networks [0.3831327965422187]
  Training of neural networks requires tedious and often manual tuning of the network architecture. We propose a systematic method to insert new layers during the training process, which eliminates the need to choose a fixed network size before training.
  arXiv  Detail & Related papers  (2023-11-27T16:44:13Z)
• WLD-Reg: A Data-dependent Within-layer Diversity Regularizer [98.78384185493624]
  Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization. We propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. We present an extensive empirical study confirming that the proposed approach enhances the performance of several state-of-the-art neural network models in multiple tasks.
  arXiv  Detail & Related papers  (2023-01-03T20:57:22Z)
• Model Fusion of Heterogeneous Neural Networks via Cross-Layer Alignment [17.735593218773758]
  We propose a novel model fusion framework, named CLAFusion, to fuse neural networks with a different number of layers. Based on the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion.
  arXiv  Detail & Related papers  (2021-10-29T05:02:23Z)
• Pruning Neural Networks with Interpolative Decompositions [5.377278489623063]
  We introduce a principled approach to neural network pruning that casts the problem as a structured low-rank matrix approximation. We demonstrate how to prune a neural network by first building a set of primitives to prune a single fully connected or convolution layer. We achieve an accuracy of 93.62 $pm$ 0.36% using VGG-16 on CIFAR-10, with a 51% FLOPS reduction.
  arXiv  Detail & Related papers  (2021-07-30T20:13:49Z)
• Backward Gradient Normalization in Deep Neural Networks [68.8204255655161]
  We introduce a new technique for gradient normalization during neural network training. The gradients are rescaled during the backward pass using normalization layers introduced at certain points within the network architecture. Results on tests with very deep neural networks show that the new technique can do an effective control of the gradient norm.
  arXiv  Detail & Related papers  (2021-06-17T13:24:43Z)
• Rethinking Skip Connection with Layer Normalization in Transformers and ResNets [49.87919454950763]
  Skip connection is a widely-used technique to improve the performance of deep neural networks. In this work, we investigate how the scale factors in the effectiveness of the skip connection.
  arXiv  Detail & Related papers  (2021-05-15T11:44:49Z)
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers [82.61117235807606]
  The success of deep neural networks in part due to the use of normalization layers. Our analysis shows how the introduction of normalization changes the landscape and can enable faster activation.
  arXiv  Detail & Related papers  (2020-06-11T23:55:54Z)
• MSE-Optimal Neural Network Initialization via Layer Fusion [68.72356718879428]
  Deep neural networks achieve state-of-the-art performance for a range of classification and inference tasks. The use of gradient combined nonvolutionity renders learning susceptible to novel problems. We propose fusing neighboring layers of deeper networks that are trained with random variables.
  arXiv  Detail & Related papers  (2020-01-28T18:25:15Z)
• Convolutional Networks with Dense Connectivity [59.30634544498946]
  We introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks.
  arXiv  Detail & Related papers  (2020-01-08T06:54:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.