Layer-adaptive sparsity for the Magnitude-based Pruning
- URL: http://arxiv.org/abs/2010.07611v2
- Date: Sun, 9 May 2021 10:19:51 GMT
- Title: Layer-adaptive sparsity for the Magnitude-based Pruning
- Authors: Jaeho Lee and Sejun Park and Sangwoo Mo and Sungsoo Ahn and Jinwoo
Shin
- Abstract summary: We propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score.
LAMP consistently outperforms popular existing schemes for layerwise sparsity selection.
- Score: 88.37510230946478
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent discoveries on neural network pruning reveal that, with a carefully
chosen layerwise sparsity, a simple magnitude-based pruning achieves
state-of-the-art tradeoff between sparsity and performance. However, without a
clear consensus on "how to choose," the layerwise sparsities are mostly
selected algorithm-by-algorithm, often resorting to handcrafted heuristics or
an extensive hyperparameter search. To fill this gap, we propose a novel
importance score for global pruning, coined layer-adaptive magnitude-based
pruning (LAMP) score; the score is a rescaled version of weight magnitude that
incorporates the model-level $\ell_2$ distortion incurred by pruning, and does
not require any hyperparameter tuning or heavy computation. Under various image
classification setups, LAMP consistently outperforms popular existing schemes
for layerwise sparsity selection. Furthermore, we observe that LAMP continues
to outperform baselines even in weight-rewinding setups, while the
connectivity-oriented layerwise sparsity (the strongest baseline overall)
performs worse than a simple global magnitude-based pruning in this case. Code:
https://github.com/jaeho-lee/layer-adaptive-sparsity
Related papers
- SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models [19.479746878680707]
Layer pruning is a potent approach to reduce network size and improve computational efficiency.
We propose a Similarity Guided fast Layer Partition pruning for compressing large deep models.
Our method outperforms the state-of-the-art methods in both accuracy and computational efficiency.
arXiv Detail & Related papers (2024-10-14T04:01:08Z) - ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language
Models [70.45441031021291]
Large Vision-Language Models (LVLMs) can understand the world comprehensively by integrating rich information from different modalities.
LVLMs are often problematic due to their massive computational/energy costs and carbon consumption.
We propose Efficient Coarse-to-Fine LayerWise Pruning (ECoFLaP), a two-stage coarse-to-fine weight pruning approach for LVLMs.
arXiv Detail & Related papers (2023-10-04T17:34:00Z) - Scaling Forward Gradient With Local Losses [117.22685584919756]
Forward learning is a biologically plausible alternative to backprop for learning deep neural networks.
We show that it is possible to substantially reduce the variance of the forward gradient by applying perturbations to activations rather than weights.
Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on ImageNet.
arXiv Detail & Related papers (2022-10-07T03:52:27Z) - Logit Attenuating Weight Normalization [5.856897366207895]
deep networks trained using gradient-based generalizations are a popular choice for solving classification and ranking problems.
Without appropriately tuned $ell$ regularization or weight decay, such networks have the tendency to make output scores (logits) and network weights large.
We propose a method called Logituating Weight Normalization (LAWN), that can be stacked onto any gradient-based generalization.
arXiv Detail & Related papers (2021-08-12T16:44:24Z) - Connectivity Matters: Neural Network Pruning Through the Lens of
Effective Sparsity [0.0]
Neural network pruning is a fruitful area of research with surging interest in high sparsity regimes.
We show that effective compression of a randomly pruned LeNet-300-100 can be orders of magnitude larger than its direct counterpart.
We develop a low-cost extension to most pruning algorithms to aim for effective, rather than direct, sparsity.
arXiv Detail & Related papers (2021-07-05T22:36:57Z) - Dynamic Probabilistic Pruning: A general framework for
hardware-constrained pruning at different granularities [80.06422693778141]
We propose a flexible new pruning mechanism that facilitates pruning at different granularities (weights, kernels, filters/feature maps)
We refer to this algorithm as Dynamic Probabilistic Pruning (DPP)
We show that DPP achieves competitive compression rates and classification accuracy when pruning common deep learning models trained on different benchmark datasets for image classification.
arXiv Detail & Related papers (2021-05-26T17:01:52Z) - DHP: Differentiable Meta Pruning via HyperNetworks [158.69345612783198]
This paper introduces a differentiable pruning method via hypernetworks for automatic network pruning.
Latent vectors control the output channels of the convolutional layers in the backbone network and act as a handle for the pruning of the layers.
Experiments are conducted on various networks for image classification, single image super-resolution, and denoising.
arXiv Detail & Related papers (2020-03-30T17:59:18Z) - Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning [83.99191569112682]
Magnitude-based pruning is one of the simplest methods for pruning neural networks.
We develop a simple pruning method, coined lookahead pruning, by extending the single layer optimization to a multi-layer optimization.
Our experimental results demonstrate that the proposed method consistently outperforms magnitude-based pruning on various networks.
arXiv Detail & Related papers (2020-02-12T05:38:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.