MINT: Deep Network Compression via Mutual Information-based Neuron Trimming

• URL: http://arxiv.org/abs/2003.08472v1
• Date: Wed, 18 Mar 2020 21:05:02 GMT
• Title: MINT: Deep Network Compression via Mutual Information-based Neuron Trimming
• Authors: Madan Ravi Ganesh, Jason J. Corso, Salimeh Yasaei Sekeh
• Abstract summary: Mutual Information-based Neuron Trimming (MINT) approaches deep compression via pruning. MINT enforces sparsity based on the strength of the relationship between filters of adjacent layers. When pruning a network, we ensure that retained filters contribute the majority of the information towards succeeding layers.
• Score: 32.449324736645586
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most approaches to deep neural network compression via pruning either evaluate a filter's importance using its weights or optimize an alternative objective function with sparsity constraints. While these methods offer a useful way to approximate contributions from similar filters, they often either ignore the dependency between layers or solve a more difficult optimization objective than standard cross-entropy. Our method, Mutual Information-based Neuron Trimming (MINT), approaches deep compression via pruning by enforcing sparsity based on the strength of the relationship between filters of adjacent layers, across every pair of layers. The relationship is calculated using conditional geometric mutual information which evaluates the amount of similar information exchanged between the filters using a graph-based criterion. When pruning a network, we ensure that retained filters contribute the majority of the information towards succeeding layers which ensures high performance. Our novel approach outperforms existing state-of-the-art compression-via-pruning methods on the standard benchmarks for this task: MNIST, CIFAR-10, and ILSVRC2012, across a variety of network architectures. In addition, we discuss our observations of a common denominator between our pruning methodology's response to adversarial attacks and calibration statistics when compared to the original network.

Related papers

• Filter Pruning via Filters Similarity in Consecutive Layers [20.29555787754269]
  Filter pruning is widely adopted to compress and accelerate the Convolutional Neural Networks (CNNs) We intuitively propose a novel pruning method by explicitly leveraging the Filters Similarity in Consecutive Layers (FSCL) Experiments demonstrate the effectiveness of FSCL, and it yields remarkable improvement over state-of-the-art on accuracy, FLOPs and parameter reduction.
  arXiv  Detail & Related papers  (2023-04-26T09:18:38Z)
• Network Pruning via Feature Shift Minimization [8.593369249204132]
  We propose a novel Feature Shift Minimization (FSM) method to compress CNN models, which evaluates the feature shift by converging the information of both features and filters. The proposed method yields state-of-the-art performance on various benchmark networks and datasets, verified by extensive experiments.
  arXiv  Detail & Related papers  (2022-07-06T12:50:26Z)
• Asymptotic Soft Cluster Pruning for Deep Neural Networks [5.311178623385279]
  Filter pruning method introduces structural sparsity by removing selected filters. We propose a novel filter pruning method called Asymptotic Soft Cluster Pruning. Our method can achieve competitive results compared with many state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2022-06-16T13:58:58Z)
• Unsharp Mask Guided Filtering [53.14430987860308]
  The goal of this paper is guided image filtering, which emphasizes the importance of structure transfer during filtering. We propose a new and simplified formulation of the guided filter inspired by unsharp masking. Our formulation enjoys a filtering prior to a low-pass filter and enables explicit structure transfer by estimating a single coefficient.
  arXiv  Detail & Related papers  (2021-06-02T19:15:34Z)
• Manifold Regularized Dynamic Network Pruning [102.24146031250034]
  This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
  arXiv  Detail & Related papers  (2021-03-10T03:59:03Z)
• SCOP: Scientific Control for Reliable Neural Network Pruning [127.20073865874636]
  This paper proposes a reliable neural network pruning algorithm by setting up a scientific control. Redundant filters can be discovered in the adversarial process of different features. Our method can reduce 57.8% parameters and 60.2% FLOPs of ResNet-101 with only 0.01% top-1 accuracy loss on ImageNet.
  arXiv  Detail & Related papers  (2020-10-21T03:02:01Z)
• ESPN: Extremely Sparse Pruned Networks [50.436905934791035]
  We show that a simple iterative mask discovery method can achieve state-of-the-art compression of very deep networks. Our algorithm represents a hybrid approach between single shot network pruning methods and Lottery-Ticket type approaches.
  arXiv  Detail & Related papers  (2020-06-28T23:09:27Z)
• Slimming Neural Networks using Adaptive Connectivity Scores [28.872080203221934]
  We propose a new single-shot, fully automated pruning algorithm called Slimming Neural networks using Adaptive Connectivity Scores (SNACS) Our proposed approach combines a probabilistic pruning framework with constraints on the underlying weight matrices. SNACS is faster by over 17x the nearest comparable method.
  arXiv  Detail & Related papers  (2020-06-22T17:45:16Z)
• Dependency Aware Filter Pruning [74.69495455411987]
  Pruning a proportion of unimportant filters is an efficient way to mitigate the inference cost. Previous work prunes filters according to their weight norms or the corresponding batch-norm scaling factors. We propose a novel mechanism to dynamically control the sparsity-inducing regularization so as to achieve the desired sparsity.
  arXiv  Detail & Related papers  (2020-05-06T07:41:22Z)
• Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression [145.04742985050808]
  We analyze two popular network compression techniques, i.e. filter pruning and low-rank decomposition, in a unified sense. By changing the way the sparsity regularization is enforced, filter pruning and low-rank decomposition can be derived accordingly. Our approach proves its potential as it compares favorably to the state-of-the-art on several benchmarks.
  arXiv  Detail & Related papers  (2020-03-19T17:57:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.