GHFP: Gradually Hard Filter Pruning

• URL: http://arxiv.org/abs/2011.03170v1
• Date: Fri, 6 Nov 2020 03:09:52 GMT
• Title: GHFP: Gradually Hard Filter Pruning
• Authors: Linhang Cai, Zhulin An, Yongjun Xu
• Abstract summary: Filter pruning is widely used to reduce the computation of deep learning, enabling the deployment of Deep Neural Networks (DNNs) in resource-limited devices. Conventional Hard Filter Pruning (HFP) method zeroizes pruned filters and stops updating them, thus reducing the search space of the model. Soft Filter Pruning (SFP) simply zeroizes pruned filters, keeping updating them in the following training epochs, thus maintaining the capacity of the network. Our question is whether SFP-based methods and HFP can be combined to achieve better performance and speed up convergence.
• Score: 9.593208961737572
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Filter pruning is widely used to reduce the computation of deep learning, enabling the deployment of Deep Neural Networks (DNNs) in resource-limited devices. Conventional Hard Filter Pruning (HFP) method zeroizes pruned filters and stops updating them, thus reducing the search space of the model. On the contrary, Soft Filter Pruning (SFP) simply zeroizes pruned filters, keeping updating them in the following training epochs, thus maintaining the capacity of the network. However, SFP, together with its variants, converges much slower than HFP due to its larger search space. Our question is whether SFP-based methods and HFP can be combined to achieve better performance and speed up convergence. Firstly, we generalize SFP-based methods and HFP to analyze their characteristics. Then we propose a Gradually Hard Filter Pruning (GHFP) method to smoothly switch from SFP-based methods to HFP during training and pruning, thus maintaining a large search space at first, gradually reducing the capacity of the model to ensure a moderate convergence speed. Experimental results on CIFAR-10/100 show that our method achieves the state-of-the-art performance.

Related papers

• CR-SFP: Learning Consistent Representation for Soft Filter Pruning [18.701621806529438]
  Soft filter pruning(SFP) has emerged as an effective pruning technique for allowing pruned filters to update and regrow to the network. We propose to mitigate this gap by learning consistent representation for soft filter pruning, dubbed as CR-SFP. CR-SFP is a simple yet effective training framework to improve the accuracy of P-NN without introducing any additional inference cost.
  arXiv  Detail & Related papers  (2023-12-17T06:41:04Z)
• An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems [10.997994515823798]
  We propose an ensemble score filter (EnSF) for solving high-dimensional nonlinear filtering problems. Unlike existing diffusion models that train neural networks to approximate the score function, we develop a training-free score estimation. EnSF provides surprising performance, compared with the state-of-the-art Local Ensemble Transform Kalman Filter method.
  arXiv  Detail & Related papers  (2023-09-02T16:48:02Z)
• Filter Pruning via Filters Similarity in Consecutive Layers [20.29555787754269]
  Filter pruning is widely adopted to compress and accelerate the Convolutional Neural Networks (CNNs) We intuitively propose a novel pruning method by explicitly leveraging the Filters Similarity in Consecutive Layers (FSCL) Experiments demonstrate the effectiveness of FSCL, and it yields remarkable improvement over state-of-the-art on accuracy, FLOPs and parameter reduction.
  arXiv  Detail & Related papers  (2023-04-26T09:18:38Z)
• Asymptotic Soft Cluster Pruning for Deep Neural Networks [5.311178623385279]
  Filter pruning method introduces structural sparsity by removing selected filters. We propose a novel filter pruning method called Asymptotic Soft Cluster Pruning. Our method can achieve competitive results compared with many state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2022-06-16T13:58:58Z)
• Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs [69.3939291118954]
  Unstructured pruning is well suited to reduce the memory footprint of convolutional neural networks (CNNs) Standard unstructured pruning (SP) reduces the memory footprint of CNNs by setting filter elements to zero. We introduce interspace pruning (IP), a general tool to improve existing pruning methods.
  arXiv  Detail & Related papers  (2022-03-15T11:50:45Z)
• Network Calculus with Flow Prolongation -- A Feedforward FIFO Analysis enabled by ML [73.11023209243326]
  Flow Prolongation (FP) has been shown to improve delay bound accuracy significantly. We introduce DeepFP, an approach to make FP scale by predicting prolongations using machine learning. DeepFP reduces delay bounds by 12.1% on average at negligible additional computational cost.
  arXiv  Detail & Related papers  (2022-02-07T08:46:47Z)
• Decoupled Dynamic Filter Networks [85.38058820176047]
  We propose the Decoupled Dynamic Filter (DDF) that can simultaneously tackle both of these shortcomings. Inspired by recent advances in attention, DDF decouples a depth-wise dynamic filter into spatial and channel dynamic filters. We observe a significant boost in performance when replacing standard convolution with DDF in classification networks.
  arXiv  Detail & Related papers  (2021-04-29T04:55:33Z)
• Manifold Regularized Dynamic Network Pruning [102.24146031250034]
  This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
  arXiv  Detail & Related papers  (2021-03-10T03:59:03Z)
• Data Agnostic Filter Gating for Efficient Deep Networks [72.4615632234314]
  Current filter pruning methods mainly leverage feature maps to generate important scores for filters and prune those with smaller scores. In this paper, we propose a data filter pruning method that uses an auxiliary network named Dagger module to induce pruning. In addition, to help prune filters with certain FLOPs constraints, we leverage an explicit FLOPs-aware regularization to directly promote pruning filters toward target FLOPs.
  arXiv  Detail & Related papers  (2020-10-28T15:26:40Z)
• Softer Pruning, Incremental Regularization [12.190136491373359]
  The Soft Filter Pruning (SFP) method zeroizes the pruned filters during training while updating them in the next training epoch. To utilize the trained pruned filters, we proposed a SofteR Filter Pruning (S RFP) method and its variant, Asymptotic SofteR Filter Pruning (AS RFP) Our methods perform well across various networks, datasets and pruning rates, also transferable to weight pruning.
  arXiv  Detail & Related papers  (2020-10-19T13:37:19Z)
• Dependency Aware Filter Pruning [74.69495455411987]
  Pruning a proportion of unimportant filters is an efficient way to mitigate the inference cost. Previous work prunes filters according to their weight norms or the corresponding batch-norm scaling factors. We propose a novel mechanism to dynamically control the sparsity-inducing regularization so as to achieve the desired sparsity.
  arXiv  Detail & Related papers  (2020-05-06T07:41:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.