Pruning On-the-Fly: A Recoverable Pruning Method without Fine-tuning
- URL: http://arxiv.org/abs/2212.12651v1
- Date: Sat, 24 Dec 2022 04:33:03 GMT
- Title: Pruning On-the-Fly: A Recoverable Pruning Method without Fine-tuning
- Authors: Dan Liu, Xue Liu
- Abstract summary: We propose a retraining-free pruning method based on hyperspherical learning and loss penalty terms.
The proposed loss penalty term pushes some of the model weights far from zero, while the rest weight values are pushed near zero.
Our proposed method can instantly recover the accuracy of a pruned model by replacing the pruned values with their mean value.
- Score: 12.90416661059601
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Most existing pruning works are resource-intensive, requiring retraining or
fine-tuning of the pruned models for accuracy. We propose a retraining-free
pruning method based on hyperspherical learning and loss penalty terms. The
proposed loss penalty term pushes some of the model weights far from zero,
while the rest weight values are pushed near zero and can be safely pruned with
no need for retraining and a negligible accuracy drop. In addition, our
proposed method can instantly recover the accuracy of a pruned model by
replacing the pruned values with their mean value. Our method obtains
state-of-the-art results in retraining-free pruning and is evaluated on
ResNet-18/50 and MobileNetV2 with ImageNet dataset. One can easily get a 50\%
pruned ResNet18 model with a 0.47\% accuracy drop. With fine-tuning, the
experiment results show that our method can significantly boost the accuracy of
the pruned models compared with existing works. For example, the accuracy of a
70\% pruned (except the first convolutional layer) MobileNetV2 model only drops
3.5\%, much less than the 7\% $\sim$ 10\% accuracy drop with conventional
methods.
Related papers
- Sr-init: An interpretable layer pruning method [11.184351630458265]
We propose a novel layer pruning method by exploring the Re-initialization.
Our SR-init method is inspired by the discovery that the accuracy drop due to re-initialization differs in various layers.
We experimentally verify the interpretability of SR-init via feature visualization.
arXiv Detail & Related papers (2023-03-14T07:26:55Z) - Gradient-Free Structured Pruning with Unlabeled Data [57.999191898036706]
We propose a gradient-free structured pruning framework that uses only unlabeled data.
Up to 40% of the original FLOP count can be reduced with less than a 4% accuracy loss across all tasks considered.
arXiv Detail & Related papers (2023-03-07T19:12:31Z) - Dynamic Sparse Training via Balancing the Exploration-Exploitation
Trade-off [19.230329532065635]
Sparse training could significantly mitigate the training costs by reducing the model size.
Existing sparse training methods mainly use either random-based or greedy-based drop-and-grow strategies.
In this work, we consider the dynamic sparse training as a sparse connectivity search problem.
Experimental results show that sparse models (up to 98% sparsity) obtained by our proposed method outperform the SOTA sparse training methods.
arXiv Detail & Related papers (2022-11-30T01:22:25Z) - CrAM: A Compression-Aware Minimizer [103.29159003723815]
We propose a new compression-aware minimizer dubbed CrAM that modifies the optimization step in a principled way.
CrAM produces dense models that can be more accurate than the standard SGD/Adam-based baselines, but which are stable under weight pruning.
CrAM can produce sparse models which perform well for transfer learning, and it also works for semi-structured 2:4 pruning patterns supported by GPU hardware.
arXiv Detail & Related papers (2022-07-28T16:13:28Z) - (Certified!!) Adversarial Robustness for Free! [116.6052628829344]
We certify 71% accuracy on ImageNet under adversarial perturbations constrained to be within a 2-norm of 0.5.
We obtain these results using only pretrained diffusion models and image classifiers, without requiring any fine tuning or retraining of model parameters.
arXiv Detail & Related papers (2022-06-21T17:27:27Z) - Paoding: Supervised Robustness-preserving Data-free Neural Network
Pruning [3.6953655494795776]
We study the neural network pruning in the emphdata-free context.
We replace the traditional aggressive one-shot strategy with a conservative one that treats the pruning as a progressive process.
Our method is implemented as a Python package named textscPaoding and evaluated with a series of experiments on diverse neural network models.
arXiv Detail & Related papers (2022-04-02T07:09:17Z) - Automatic Neural Network Pruning that Efficiently Preserves the Model
Accuracy [2.538209532048867]
pruning filters is a common solution, but most existing pruning methods do not preserve the model accuracy efficiently.
We propose an automatic pruning method that learns which neurons to preserve in order to maintain the model accuracy while reducing the FLOPs to a predefined target.
We achieve a 52.00% FLOPs reduction on ResNet-50, with a Top-1 accuracy of 47.51% after pruning and a state-of-the-art (SOTA) accuracy of 76.63% after finetuning.
arXiv Detail & Related papers (2021-11-18T11:29:35Z) - Post-training deep neural network pruning via layer-wise calibration [70.65691136625514]
We propose a data-free extension of the approach for computer vision models based on automatically-generated synthetic fractal images.
When using real data, we are able to get a ResNet50 model on ImageNet with 65% sparsity rate in 8-bit precision in a post-training setting.
arXiv Detail & Related papers (2021-04-30T14:20:51Z) - ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting [105.97936163854693]
We propose ResRep, which slims down a CNN by reducing the width (number of output channels) of convolutional layers.
Inspired by the neurobiology research about the independence of remembering and forgetting, we propose to re- parameterize a CNN into the remembering parts and forgetting parts.
We equivalently merge the remembering and forgetting parts into the original architecture with narrower layers.
arXiv Detail & Related papers (2020-07-07T07:56:45Z) - Movement Pruning: Adaptive Sparsity by Fine-Tuning [115.91907953454034]
Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning.
We propose the use of movement pruning, a simple, deterministic first-order weight pruning method.
Experiments show that when pruning large pretrained language models, movement pruning shows significant improvements in high-sparsity regimes.
arXiv Detail & Related papers (2020-05-15T17:54:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.