Network Pruning via Resource Reallocation
- URL: http://arxiv.org/abs/2103.01847v1
- Date: Tue, 2 Mar 2021 16:28:10 GMT
- Title: Network Pruning via Resource Reallocation
- Authors: Yuenan Hou, Zheng Ma, Chunxiao Liu, Zhe Wang, and Chen Change Loy
- Abstract summary: We propose a simple yet effective channel pruning technique, termed network Pruning via rEsource rEalLocation (PEEL)
PEEL first constructs a predefined backbone and then conducts resource reallocation on it to shift parameters from less informative layers to more important layers in one round.
Experimental results show that structures uncovered by PEEL exhibit competitive performance with state-of-the-art pruning algorithms under various pruning settings.
- Score: 75.85066435085595
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Channel pruning is broadly recognized as an effective approach to obtain a
small compact model through eliminating unimportant channels from a large
cumbersome network. Contemporary methods typically perform iterative pruning
procedure from the original over-parameterized model, which is both tedious and
expensive especially when the pruning is aggressive. In this paper, we propose
a simple yet effective channel pruning technique, termed network Pruning via
rEsource rEalLocation (PEEL), to quickly produce a desired slim model with
negligible cost. Specifically, PEEL first constructs a predefined backbone and
then conducts resource reallocation on it to shift parameters from less
informative layers to more important layers in one round, thus amplifying the
positive effect of these informative layers. To demonstrate the effectiveness
of PEEL , we perform extensive experiments on ImageNet with ResNet-18,
ResNet-50, MobileNetV2, MobileNetV3-small and EfficientNet-B0. Experimental
results show that structures uncovered by PEEL exhibit competitive performance
with state-of-the-art pruning algorithms under various pruning settings. Our
code is available at https://github.com/cardwing/Codes-for-PEEL.
Related papers
- Dynamic Structure Pruning for Compressing CNNs [13.73717878732162]
We introduce a novel structure pruning method, termed as dynamic structure pruning, to identify optimal pruning granularities for intra-channel pruning.
The experimental results show that dynamic structure pruning achieves state-of-the-art pruning performance and better realistic acceleration on a GPU compared with channel pruning.
arXiv Detail & Related papers (2023-03-17T02:38:53Z) - Visual Saliency-Guided Channel Pruning for Deep Visual Detectors in
Autonomous Driving [3.236217153362305]
Deep neural network (DNN) pruning has become a de facto component for deploying on resource-constrained devices.
We propose a novel gradient-based saliency measure for visual detection and use it to guide our channel pruning.
Experiments on the KITTI and COCO traffic datasets demonstrate our pruning method's efficacy and superiority over state-of-the-art competing approaches.
arXiv Detail & Related papers (2023-03-04T22:08:22Z) - CATRO: Channel Pruning via Class-Aware Trace Ratio Optimization [61.71504948770445]
We propose a novel channel pruning method via Class-Aware Trace Ratio Optimization (CATRO) to reduce the computational burden and accelerate the model inference.
We show that CATRO achieves higher accuracy with similar cost or lower cost with similar accuracy than other state-of-the-art channel pruning algorithms.
Because of its class-aware property, CATRO is suitable to prune efficient networks adaptively for various classification subtasks, enhancing handy deployment and usage of deep networks in real-world applications.
arXiv Detail & Related papers (2021-10-21T06:26:31Z) - AdaPruner: Adaptive Channel Pruning and Effective Weights Inheritance [9.3421559369389]
We propose a pruning framework that adaptively determines the number of each layer's channels as well as the wights inheritance criteria for sub-network.
AdaPruner allows to obtain pruned network quickly, accurately and efficiently.
On ImageNet, we reduce 32.8% FLOPs of MobileNetV2 with only 0.62% decrease for top-1 accuracy, which exceeds all previous state-of-the-art channel pruning methods.
arXiv Detail & Related papers (2021-09-14T01:52:05Z) - Connectivity Matters: Neural Network Pruning Through the Lens of
Effective Sparsity [0.0]
Neural network pruning is a fruitful area of research with surging interest in high sparsity regimes.
We show that effective compression of a randomly pruned LeNet-300-100 can be orders of magnitude larger than its direct counterpart.
We develop a low-cost extension to most pruning algorithms to aim for effective, rather than direct, sparsity.
arXiv Detail & Related papers (2021-07-05T22:36:57Z) - BCNet: Searching for Network Width with Bilaterally Coupled Network [56.14248440683152]
We introduce a new supernet called Bilaterally Coupled Network (BCNet) to address this issue.
In BCNet, each channel is fairly trained and responsible for the same amount of network widths, thus each network width can be evaluated more accurately.
Our method achieves state-of-the-art or competing performance over other baseline methods.
arXiv Detail & Related papers (2021-05-21T18:54:03Z) - Neural Pruning via Growing Regularization [82.9322109208353]
We extend regularization to tackle two central problems of pruning: pruning schedule and weight importance scoring.
Specifically, we propose an L2 regularization variant with rising penalty factors and show it can bring significant accuracy gains.
The proposed algorithms are easy to implement and scalable to large datasets and networks in both structured and unstructured pruning.
arXiv Detail & Related papers (2020-12-16T20:16:28Z) - Go Wide, Then Narrow: Efficient Training of Deep Thin Networks [62.26044348366186]
We propose an efficient method to train a deep thin network with a theoretic guarantee.
By training with our method, ResNet50 can outperform ResNet101, and BERT Base can be comparable with BERT Large.
arXiv Detail & Related papers (2020-07-01T23:34:35Z) - PruneNet: Channel Pruning via Global Importance [22.463154358632472]
We propose a simple-yet-effective method for pruning channels based on a computationally light-weight yet effective data driven optimization step.
With non-uniform pruning across the layers on ResNet-$50$, we are able to match the FLOP reduction of state-of-the-art channel pruning results.
arXiv Detail & Related papers (2020-05-22T17:09:56Z) - Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning [83.99191569112682]
Magnitude-based pruning is one of the simplest methods for pruning neural networks.
We develop a simple pruning method, coined lookahead pruning, by extending the single layer optimization to a multi-layer optimization.
Our experimental results demonstrate that the proposed method consistently outperforms magnitude-based pruning on various networks.
arXiv Detail & Related papers (2020-02-12T05:38:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.