Sr-init: An interpretable layer pruning method
- URL: http://arxiv.org/abs/2303.07677v1
- Date: Tue, 14 Mar 2023 07:26:55 GMT
- Title: Sr-init: An interpretable layer pruning method
- Authors: Hui Tang, Yao Lu, Qi Xuan
- Abstract summary: We propose a novel layer pruning method by exploring the Re-initialization.
Our SR-init method is inspired by the discovery that the accuracy drop due to re-initialization differs in various layers.
We experimentally verify the interpretability of SR-init via feature visualization.
- Score: 11.184351630458265
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Despite the popularization of deep neural networks (DNNs) in many fields, it
is still challenging to deploy state-of-the-art models to resource-constrained
devices due to high computational overhead. Model pruning provides a feasible
solution to the aforementioned challenges. However, the interpretation of
existing pruning criteria is always overlooked. To counter this issue, we
propose a novel layer pruning method by exploring the Stochastic
Re-initialization. Our SR-init method is inspired by the discovery that the
accuracy drop due to stochastic re-initialization of layer parameters differs
in various layers. On the basis of this observation, we come up with a layer
pruning criterion, i.e., those layers that are not sensitive to stochastic
re-initialization (low accuracy drop) produce less contribution to the model
and could be pruned with acceptable loss. Afterward, we experimentally verify
the interpretability of SR-init via feature visualization. The visual
explanation demonstrates that SR-init is theoretically feasible, thus we
compare it with state-of-the-art methods to further evaluate its
practicability. As for ResNet56 on CIFAR-10 and CIFAR-100, SR-init achieves a
great reduction in parameters (63.98% and 37.71%) with an ignorable drop in
top-1 accuracy (-0.56% and 0.8%). With ResNet50 on ImageNet, we achieve a
15.59% FLOPs reduction by removing 39.29% of the parameters, with only a drop
of 0.6% in top-1 accuracy. Our code is available at
https://github.com/huitang-zjut/SRinit.
Related papers
- Towards Generalized Entropic Sparsification for Convolutional Neural Networks [0.0]
Convolutional neural networks (CNNs) are reported to be overparametrized.
Here, we introduce a layer-by-layer data-driven pruning method based on the mathematical idea aiming at a computationally-scalable entropic relaxation of the pruning problem.
The sparse subnetwork is found from the pre-trained (full) CNN using the network entropy minimization as a sparsity constraint.
arXiv Detail & Related papers (2024-04-06T21:33:39Z) - Pruning On-the-Fly: A Recoverable Pruning Method without Fine-tuning [12.90416661059601]
We propose a retraining-free pruning method based on hyperspherical learning and loss penalty terms.
The proposed loss penalty term pushes some of the model weights far from zero, while the rest weight values are pushed near zero.
Our proposed method can instantly recover the accuracy of a pruned model by replacing the pruned values with their mean value.
arXiv Detail & Related papers (2022-12-24T04:33:03Z) - (Certified!!) Adversarial Robustness for Free! [116.6052628829344]
We certify 71% accuracy on ImageNet under adversarial perturbations constrained to be within a 2-norm of 0.5.
We obtain these results using only pretrained diffusion models and image classifiers, without requiring any fine tuning or retraining of model parameters.
arXiv Detail & Related papers (2022-06-21T17:27:27Z) - End-to-End Sensitivity-Based Filter Pruning [49.61707925611295]
We present a sensitivity-based filter pruning algorithm (SbF-Pruner) to learn the importance scores of filters of each layer end-to-end.
Our method learns the scores from the filter weights, enabling it to account for the correlations between the filters of each layer.
arXiv Detail & Related papers (2022-04-15T10:21:05Z) - Toward Compact Deep Neural Networks via Energy-Aware Pruning [2.578242050187029]
We propose a novel energy-aware pruning method that quantifies the importance of each filter in the network using nuclear-norm (NN)
We achieve competitive results with 40.4/49.8% of FLOPs and 45.9/52.9% of parameter reduction with 94.13/94.61% in the Top-1 accuracy with ResNet-56/110 on CIFAR-10.
arXiv Detail & Related papers (2021-03-19T15:33:16Z) - Hessian-Aware Pruning and Optimal Neural Implant [74.3282611517773]
Pruning is an effective method to reduce the memory footprint and FLOPs associated with neural network models.
We introduce a new Hessian Aware Pruning method coupled with a Neural Implant approach that uses second-order sensitivity as a metric for structured pruning.
arXiv Detail & Related papers (2021-01-22T04:08:03Z) - Non-Parametric Adaptive Network Pruning [125.4414216272874]
We introduce non-parametric modeling to simplify the algorithm design.
Inspired by the face recognition community, we use a message passing algorithm to obtain an adaptive number of exemplars.
EPruner breaks the dependency on the training data in determining the "important" filters.
arXiv Detail & Related papers (2021-01-20T06:18:38Z) - Layer Pruning via Fusible Residual Convolutional Block for Deep Neural
Networks [15.64167076052513]
layer pruning has less inference time and runtime memory usage when the same FLOPs and number of parameters are pruned.
We propose a simple layer pruning method using residual convolutional block (ResConv)
Our pruning method achieves excellent performance of compression and acceleration over the state-thearts on different datasets.
arXiv Detail & Related papers (2020-11-29T12:51:16Z) - ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting [105.97936163854693]
We propose ResRep, which slims down a CNN by reducing the width (number of output channels) of convolutional layers.
Inspired by the neurobiology research about the independence of remembering and forgetting, we propose to re- parameterize a CNN into the remembering parts and forgetting parts.
We equivalently merge the remembering and forgetting parts into the original architecture with narrower layers.
arXiv Detail & Related papers (2020-07-07T07:56:45Z) - HRank: Filter Pruning using High-Rank Feature Map [149.86903824840752]
We propose a novel filter pruning method by exploring the High Rank of feature maps (HRank)
Our HRank is inspired by the discovery that the average rank of multiple feature maps generated by a single filter is always the same.
Based on HRank, we develop a method that is mathematically formulated to prune filters with low-rank feature maps.
arXiv Detail & Related papers (2020-02-24T11:50:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.