A Feature-map Discriminant Perspective for Pruning Deep Neural Networks

• URL: http://arxiv.org/abs/2005.13796v1
• Date: Thu, 28 May 2020 06:25:22 GMT
• Title: A Feature-map Discriminant Perspective for Pruning Deep Neural Networks
• Authors: Zejiang Hou and Sun-Yuan Kung
• Abstract summary: We present a new mathematical formulation to accurately and efficiently quantify the feature-map discriminativeness. We analyze the theoretical property of DI, specifically the non-decreasing property, that makes DI a valid selection criterion. We propose a DI-based greedy pruning algorithm and structure distillation technique to automatically decide the pruned structure.
• Score: 24.062226363823257
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Network pruning has become the de facto tool to accelerate deep neural networks for mobile and edge applications. Recently, feature-map discriminant based channel pruning has shown promising results, as it aligns well with the CNN objective of differentiating multiple classes and offers better interpretability of the pruning decision. However, existing discriminant-based methods are challenged by computation inefficiency, as there is a lack of theoretical guidance on quantifying the feature-map discriminant power. In this paper, we present a new mathematical formulation to accurately and efficiently quantify the feature-map discriminativeness, which gives rise to a novel criterion,Discriminant Information(DI). We analyze the theoretical property of DI, specifically the non-decreasing property, that makes DI a valid selection criterion. DI-based pruning removes channels with minimum influence to DI value, as they contain little information regarding to the discriminant power. The versatility of DI criterion also enables an intra-layer mixed precision quantization to further compress the network. Moreover, we propose a DI-based greedy pruning algorithm and structure distillation technique to automatically decide the pruned structure that satisfies certain resource budget, which is a common requirement in reality. Extensive experiments demonstratethe effectiveness of our method: our pruned ResNet50 on ImageNet achieves 44% FLOPs reduction without any Top-1 accuracy loss compared to unpruned model

Related papers

• Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regression [7.450181695527364]
  We study the properties of differentiable neural networks activated by rectified power unit (RePU) functions. We establish error bounds for simultaneously approximating $Cs$ smooth functions and their derivatives using RePU-activated deep neural networks.
  arXiv  Detail & Related papers  (2023-05-01T00:09:48Z)
• Binarizing Sparse Convolutional Networks for Efficient Point Cloud Analysis [93.55896765176414]
  We propose binary sparse convolutional networks called BSC-Net for efficient point cloud analysis. We employ the differentiable search strategies to discover the optimal opsitions for active site matching in the shifted sparse convolution. Our BSC-Net achieves significant improvement upon our srtong baseline and outperforms the state-of-the-art network binarization methods.
  arXiv  Detail & Related papers  (2023-03-27T13:47:06Z)
• The Cascaded Forward Algorithm for Neural Network Training [61.06444586991505]
  We propose a new learning framework for neural networks, namely Cascaded Forward (CaFo) algorithm, which does not rely on BP optimization as that in FF. Unlike FF, our framework directly outputs label distributions at each cascaded block, which does not require generation of additional negative samples. In our framework each block can be trained independently, so it can be easily deployed into parallel acceleration systems.
  arXiv  Detail & Related papers  (2023-03-17T02:01:11Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• Efficient Stein Variational Inference for Reliable Distribution-lossless Network Pruning [23.22021752821507]
  We propose a novel distribution-lossless pruning method, named vanillaP, to theoretically find the pruned lottery within Bayesian treatment. Our method can obtain sparser networks with great performance while providing quantified reliability for the pruned model.
  arXiv  Detail & Related papers  (2022-12-07T09:31:47Z)
• SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance [37.82255888371488]
  We propose a novel integrated gradient pruning criterion, in which the relevance of each neuron is defined as the integral of the gradient variation on a path towards this neuron removal. We show through extensive validation on several datasets, architectures as well as pruning scenarios that the proposed method, dubbed SInGE, significantly outperforms existing state-of-the-art pruning methods.
  arXiv  Detail & Related papers  (2022-07-08T18:27:42Z)
• Can pruning improve certified robustness of neural networks? [106.03070538582222]
  We show that neural network pruning can improve empirical robustness of deep neural networks (NNs) Our experiments show that by appropriately pruning an NN, its certified accuracy can be boosted up to 8.2% under standard training. We additionally observe the existence of certified lottery tickets that can match both standard and certified robust accuracies of the original dense models.
  arXiv  Detail & Related papers  (2022-06-15T05:48:51Z)
• CATRO: Channel Pruning via Class-Aware Trace Ratio Optimization [61.71504948770445]
  We propose a novel channel pruning method via Class-Aware Trace Ratio Optimization (CATRO) to reduce the computational burden and accelerate the model inference. We show that CATRO achieves higher accuracy with similar cost or lower cost with similar accuracy than other state-of-the-art channel pruning algorithms. Because of its class-aware property, CATRO is suitable to prune efficient networks adaptively for various classification subtasks, enhancing handy deployment and usage of deep networks in real-world applications.
  arXiv  Detail & Related papers  (2021-10-21T06:26:31Z)
• Neural Network Pruning Through Constrained Reinforcement Learning [3.2880869992413246]
  We propose a general methodology for pruning neural networks. Our proposed methodology can prune neural networks to respect pre-defined computational budgets. We prove the effectiveness of our approach via comparison with state-of-the-art methods on standard image classification datasets.
  arXiv  Detail & Related papers  (2021-10-16T11:57:38Z)
• AIP: Adversarial Iterative Pruning Based on Knowledge Transfer for Convolutional Neural Networks [7.147985297123097]
  convolutional neural networks (CNNs) take a fair amount of computation cost. Current pruning methods can compress CNNs with little performance drop, but when the pruning ratio increases, the accuracy loss is more serious. We propose a novel adversarial iterative pruning method (AIP) for CNNs based on knowledge transfer.
  arXiv  Detail & Related papers  (2021-08-31T02:38:36Z)
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search [55.164053971213576]
  convolutional neural network has achieved great success in fulfilling computer vision tasks despite large computation overhead. Structured (channel) pruning is usually applied to reduce the model redundancy while preserving the network structure. Existing structured pruning methods require hand-crafted rules which may lead to tremendous pruning space.
  arXiv  Detail & Related papers  (2020-11-04T07:43:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.