Softer Pruning, Incremental Regularization
- URL: http://arxiv.org/abs/2010.09498v1
- Date: Mon, 19 Oct 2020 13:37:19 GMT
- Title: Softer Pruning, Incremental Regularization
- Authors: Linhang Cai, Zhulin An, Chuanguang Yang and Yongjun Xu
- Abstract summary: The Soft Filter Pruning (SFP) method zeroizes the pruned filters during training while updating them in the next training epoch.
To utilize the trained pruned filters, we proposed a SofteR Filter Pruning (S RFP) method and its variant, Asymptotic SofteR Filter Pruning (AS RFP)
Our methods perform well across various networks, datasets and pruning rates, also transferable to weight pruning.
- Score: 12.190136491373359
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Network pruning is widely used to compress Deep Neural Networks (DNNs). The
Soft Filter Pruning (SFP) method zeroizes the pruned filters during training
while updating them in the next training epoch. Thus the trained information of
the pruned filters is completely dropped. To utilize the trained pruned
filters, we proposed a SofteR Filter Pruning (SRFP) method and its variant,
Asymptotic SofteR Filter Pruning (ASRFP), simply decaying the pruned weights
with a monotonic decreasing parameter. Our methods perform well across various
networks, datasets and pruning rates, also transferable to weight pruning. On
ILSVRC-2012, ASRFP prunes 40% of the parameters on ResNet-34 with 1.63% top-1
and 0.68% top-5 accuracy improvement. In theory, SRFP and ASRFP are an
incremental regularization of the pruned filters. Besides, We note that SRFP
and ASRFP pursue better results while slowing down the speed of convergence.
Related papers
- CR-SFP: Learning Consistent Representation for Soft Filter Pruning [18.701621806529438]
Soft filter pruning(SFP) has emerged as an effective pruning technique for allowing pruned filters to update and regrow to the network.
We propose to mitigate this gap by learning consistent representation for soft filter pruning, dubbed as CR-SFP.
CR-SFP is a simple yet effective training framework to improve the accuracy of P-NN without introducing any additional inference cost.
arXiv Detail & Related papers (2023-12-17T06:41:04Z) - End-to-End Sensitivity-Based Filter Pruning [49.61707925611295]
We present a sensitivity-based filter pruning algorithm (SbF-Pruner) to learn the importance scores of filters of each layer end-to-end.
Our method learns the scores from the filter weights, enabling it to account for the correlations between the filters of each layer.
arXiv Detail & Related papers (2022-04-15T10:21:05Z) - Interspace Pruning: Using Adaptive Filter Representations to Improve
Training of Sparse CNNs [69.3939291118954]
Unstructured pruning is well suited to reduce the memory footprint of convolutional neural networks (CNNs)
Standard unstructured pruning (SP) reduces the memory footprint of CNNs by setting filter elements to zero.
We introduce interspace pruning (IP), a general tool to improve existing pruning methods.
arXiv Detail & Related papers (2022-03-15T11:50:45Z) - Pruning Networks with Cross-Layer Ranking & k-Reciprocal Nearest Filters [151.2423480789271]
A novel pruning method, termed CLR-RNF, is proposed for filter-level network pruning.
We conduct image classification on CIFAR-10 and ImageNet to demonstrate the superiority of our CLR-RNF over the state-of-the-arts.
arXiv Detail & Related papers (2022-02-15T04:53:24Z) - Non-Parametric Adaptive Network Pruning [125.4414216272874]
We introduce non-parametric modeling to simplify the algorithm design.
Inspired by the face recognition community, we use a message passing algorithm to obtain an adaptive number of exemplars.
EPruner breaks the dependency on the training data in determining the "important" filters.
arXiv Detail & Related papers (2021-01-20T06:18:38Z) - GHFP: Gradually Hard Filter Pruning [9.593208961737572]
Filter pruning is widely used to reduce the computation of deep learning, enabling the deployment of Deep Neural Networks (DNNs) in resource-limited devices.
Conventional Hard Filter Pruning (HFP) method zeroizes pruned filters and stops updating them, thus reducing the search space of the model.
Soft Filter Pruning (SFP) simply zeroizes pruned filters, keeping updating them in the following training epochs, thus maintaining the capacity of the network.
Our question is whether SFP-based methods and HFP can be combined to achieve better performance and speed up convergence.
arXiv Detail & Related papers (2020-11-06T03:09:52Z) - Data Agnostic Filter Gating for Efficient Deep Networks [72.4615632234314]
Current filter pruning methods mainly leverage feature maps to generate important scores for filters and prune those with smaller scores.
In this paper, we propose a data filter pruning method that uses an auxiliary network named Dagger module to induce pruning.
In addition, to help prune filters with certain FLOPs constraints, we leverage an explicit FLOPs-aware regularization to directly promote pruning filters toward target FLOPs.
arXiv Detail & Related papers (2020-10-28T15:26:40Z) - Filter Sketch for Network Pruning [184.41079868885265]
We propose a novel network pruning approach by information preserving of pre-trained network weights (filters)
Our approach, referred to as FilterSketch, encodes the second-order information of pre-trained weights.
Experiments on CIFAR-10 show that FilterSketch reduces 63.3% of FLOPs and prunes 59.9% of network parameters with negligible accuracy cost.
arXiv Detail & Related papers (2020-01-23T13:57:08Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.