CHEX: CHannel EXploration for CNN Model Compression
- URL: http://arxiv.org/abs/2203.15794v1
- Date: Tue, 29 Mar 2022 17:52:41 GMT
- Title: CHEX: CHannel EXploration for CNN Model Compression
- Authors: Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma, Kun Yuan, Yi Xu,
Yen-Kuang Chen, Rong Jin, Yuan Xie, Sun-Yuan Kung
- Abstract summary: We propose a novel Channel Exploration methodology, dubbed as CHEX, to rectify these problems.
CheX repeatedly prunes and regrows the channels throughout the training process, which reduces the risk of pruning important channels prematurely.
Results demonstrate that CHEX can effectively reduce the FLOPs of diverse CNN architectures on a variety of computer vision tasks.
- Score: 47.3520447163165
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Channel pruning has been broadly recognized as an effective technique to
reduce the computation and memory cost of deep convolutional neural networks.
However, conventional pruning methods have limitations in that: they are
restricted to pruning process only, and they require a fully pre-trained large
model. Such limitations may lead to sub-optimal model quality as well as
excessive memory and training cost. In this paper, we propose a novel Channel
Exploration methodology, dubbed as CHEX, to rectify these problems. As opposed
to pruning-only strategy, we propose to repeatedly prune and regrow the
channels throughout the training process, which reduces the risk of pruning
important channels prematurely. More exactly: From intra-layer's aspect, we
tackle the channel pruning problem via a well known column subset selection
(CSS) formulation. From inter-layer's aspect, our regrowing stages open a path
for dynamically re-allocating the number of channels across all the layers
under a global channel sparsity constraint. In addition, all the exploration
process is done in a single training from scratch without the need of a
pre-trained large model. Experimental results demonstrate that CHEX can
effectively reduce the FLOPs of diverse CNN architectures on a variety of
computer vision tasks, including image classification, object detection,
instance segmentation, and 3D vision. For example, our compressed ResNet-50
model on ImageNet dataset achieves 76% top1 accuracy with only 25% FLOPs of the
original ResNet-50 model, outperforming previous state-of-the-art channel
pruning methods. The checkpoints and code are available at here .
Related papers
- Pruning Very Deep Neural Network Channels for Efficient Inference [6.497816402045099]
Given a trained CNN model, we propose an iterative two-step algorithm to effectively prune each layer.
VGG-16 achieves the state-of-the-art results by 5x speed-up along with only 0.3% increase of error.
Our method is able to accelerate modern networks like ResNet, Xception and suffers only 1.4%, 1.0% accuracy loss under 2x speed-up respectively.
arXiv Detail & Related papers (2022-11-14T06:48:33Z) - Interpretations Steered Network Pruning via Amortized Inferred Saliency
Maps [85.49020931411825]
Convolutional Neural Networks (CNNs) compression is crucial to deploying these models in edge devices with limited resources.
We propose to address the channel pruning problem from a novel perspective by leveraging the interpretations of a model to steer the pruning process.
We tackle this challenge by introducing a selector model that predicts real-time smooth saliency masks for pruned models.
arXiv Detail & Related papers (2022-09-07T01:12:11Z) - Structured Pruning is All You Need for Pruning CNNs at Initialization [38.88730369884401]
Pruning is a popular technique for reducing the model size and computational cost of convolutional neural networks (CNNs)
We propose PreCropping, a structured hardware-efficient model compression scheme.
Compared to weight pruning, the proposed scheme is regular and dense in both storage and computation without sacrificing accuracy.
arXiv Detail & Related papers (2022-03-04T19:54:31Z) - AdaPruner: Adaptive Channel Pruning and Effective Weights Inheritance [9.3421559369389]
We propose a pruning framework that adaptively determines the number of each layer's channels as well as the wights inheritance criteria for sub-network.
AdaPruner allows to obtain pruned network quickly, accurately and efficiently.
On ImageNet, we reduce 32.8% FLOPs of MobileNetV2 with only 0.62% decrease for top-1 accuracy, which exceeds all previous state-of-the-art channel pruning methods.
arXiv Detail & Related papers (2021-09-14T01:52:05Z) - GDP: Stabilized Neural Network Pruning via Gates with Differentiable
Polarization [84.57695474130273]
Gate-based or importance-based pruning methods aim to remove channels whose importance is smallest.
GDP can be plugged before convolutional layers without bells and whistles, to control the on-and-off of each channel.
Experiments conducted over CIFAR-10 and ImageNet datasets show that the proposed GDP achieves the state-of-the-art performance.
arXiv Detail & Related papers (2021-09-06T03:17:10Z) - Group Fisher Pruning for Practical Network Compression [58.25776612812883]
We present a general channel pruning approach that can be applied to various complicated structures.
We derive a unified metric based on Fisher information to evaluate the importance of a single channel and coupled channels.
Our method can be used to prune any structures including those with coupled channels.
arXiv Detail & Related papers (2021-08-02T08:21:44Z) - Channel Scaling: A Scale-and-Select Approach for Transfer Learning [2.6304695993930594]
Transfer learning with pre-trained neural networks is a common strategy for training classifiers in medical image analysis.
We propose a novel approach to efficiently build small and well performing networks by introducing the channel-scaling layers.
By imposing L1 regularization and thresholding on the scaling weights, this framework iteratively removes unnecessary feature channels from a pre-trained model.
arXiv Detail & Related papers (2021-03-22T23:26:57Z) - Network Pruning via Resource Reallocation [75.85066435085595]
We propose a simple yet effective channel pruning technique, termed network Pruning via rEsource rEalLocation (PEEL)
PEEL first constructs a predefined backbone and then conducts resource reallocation on it to shift parameters from less informative layers to more important layers in one round.
Experimental results show that structures uncovered by PEEL exhibit competitive performance with state-of-the-art pruning algorithms under various pruning settings.
arXiv Detail & Related papers (2021-03-02T16:28:10Z) - UCP: Uniform Channel Pruning for Deep Convolutional Neural Networks
Compression and Acceleration [24.42067007684169]
We propose a novel uniform channel pruning (UCP) method to prune deep CNN.
The unimportant channels, including convolutional kernels related to them, are pruned directly.
We verify our method on CIFAR-10, CIFAR-100 and ILSVRC-2012 for image classification.
arXiv Detail & Related papers (2020-10-03T01:51:06Z) - ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting [105.97936163854693]
We propose ResRep, which slims down a CNN by reducing the width (number of output channels) of convolutional layers.
Inspired by the neurobiology research about the independence of remembering and forgetting, we propose to re- parameterize a CNN into the remembering parts and forgetting parts.
We equivalently merge the remembering and forgetting parts into the original architecture with narrower layers.
arXiv Detail & Related papers (2020-07-07T07:56:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.