Feather: An Elegant Solution to Effective DNN Sparsification
- URL: http://arxiv.org/abs/2310.02448v1
- Date: Tue, 3 Oct 2023 21:37:13 GMT
- Title: Feather: An Elegant Solution to Effective DNN Sparsification
- Authors: Athanasios Glentis Georgoulakis, George Retsinas, Petros Maragos
- Abstract summary: Feather is an efficient sparse training module utilizing the powerful Straight-Through Estimator as its core.
It achieves state-of-the-art Top-1 validation accuracy using the ResNet-50 architecture.
- Score: 26.621226121259372
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neural Network pruning is an increasingly popular way for producing compact
and efficient models, suitable for resource-limited environments, while
preserving high performance. While the pruning can be performed using a
multi-cycle training and fine-tuning process, the recent trend is to encompass
the sparsification process during the standard course of training. To this end,
we introduce Feather, an efficient sparse training module utilizing the
powerful Straight-Through Estimator as its core, coupled with a new
thresholding operator and a gradient scaling technique, enabling robust,
out-of-the-box sparsification performance. Feather's effectiveness and
adaptability is demonstrated using various architectures on the CIFAR dataset,
while on ImageNet it achieves state-of-the-art Top-1 validation accuracy using
the ResNet-50 architecture, surpassing existing methods, including more complex
and computationally heavy ones, by a considerable margin. Code is publicly
available at https://github.com/athglentis/feather .
Related papers
- Block Pruning for Enhanced Efficiency in Convolutional Neural Networks [7.110116320545541]
This paper presents a novel approach to network pruning, targeting block pruning in deep neural networks for edge computing environments.
Our method diverges from traditional techniques that utilize proxy metrics, instead employing a direct block removal strategy to assess the impact on classification accuracy.
arXiv Detail & Related papers (2023-12-28T08:54:48Z) - Not All Data Matters: An End-to-End Adaptive Dataset Pruning Framework
for Enhancing Model Performance and Efficiency [9.460023981858319]
We propose an end-to-end Adaptive DAtaset PRUNing framework called AdaPruner.
AdaPruner iteratively prunes redundant samples to an expected pruning ratio.
It can still significantly enhance model performance even after pruning up to 10-30% of the training data.
arXiv Detail & Related papers (2023-12-09T16:01:21Z) - Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep
Neural Networks [48.089501687522954]
We propose a novel layer-adaptive weight-pruning approach for Deep Neural Networks (DNNs)
Our approach takes into account the collective influence of all layers to design a layer-adaptive pruning scheme.
Our experiments demonstrate the superiority of our approach over existing methods on the ImageNet and CIFAR-10 datasets.
arXiv Detail & Related papers (2023-08-21T03:22:47Z) - Fast as CHITA: Neural Network Pruning with Combinatorial Optimization [9.440450886684603]
We propose a novel optimization-based pruning framework that considers the combined effect of pruning (and updating) multiple weights subject to a sparsity constraint.
Our approach, CHITA, extends the classical Brain Surgeon framework and results in significant improvements in speed, memory, and performance.
arXiv Detail & Related papers (2023-02-28T15:03:18Z) - Pushing the Efficiency Limit Using Structured Sparse Convolutions [82.31130122200578]
We propose Structured Sparse Convolution (SSC), which leverages the inherent structure in images to reduce the parameters in the convolutional filter.
We show that SSC is a generalization of commonly used layers (depthwise, groupwise and pointwise convolution) in efficient architectures''
Architectures based on SSC achieve state-of-the-art performance compared to baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet classification benchmarks.
arXiv Detail & Related papers (2022-10-23T18:37:22Z) - Efficient Few-Shot Object Detection via Knowledge Inheritance [62.36414544915032]
Few-shot object detection (FSOD) aims at learning a generic detector that can adapt to unseen tasks with scarce training samples.
We present an efficient pretrain-transfer framework (PTF) baseline with no computational increment.
We also propose an adaptive length re-scaling (ALR) strategy to alleviate the vector length inconsistency between the predicted novel weights and the pretrained base weights.
arXiv Detail & Related papers (2022-03-23T06:24:31Z) - Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models.
Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely.
Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
arXiv Detail & Related papers (2021-10-01T10:03:57Z) - Top-KAST: Top-K Always Sparse Training [50.05611544535801]
We propose Top-KAST, a method that preserves constant sparsity throughout training.
We show that it performs comparably to or better than previous works when training models on the established ImageNet benchmark.
In addition to our ImageNet results, we also demonstrate our approach in the domain of language modeling.
arXiv Detail & Related papers (2021-06-07T11:13:05Z) - Network Pruning via Resource Reallocation [75.85066435085595]
We propose a simple yet effective channel pruning technique, termed network Pruning via rEsource rEalLocation (PEEL)
PEEL first constructs a predefined backbone and then conducts resource reallocation on it to shift parameters from less informative layers to more important layers in one round.
Experimental results show that structures uncovered by PEEL exhibit competitive performance with state-of-the-art pruning algorithms under various pruning settings.
arXiv Detail & Related papers (2021-03-02T16:28:10Z) - An Image Enhancing Pattern-based Sparsity for Real-time Inference on
Mobile Devices [58.62801151916888]
We introduce a new sparsity dimension, namely pattern-based sparsity that comprises pattern and connectivity sparsity, and becoming both highly accurate and hardware friendly.
Our approach on the new pattern-based sparsity naturally fits into compiler optimization for highly efficient DNN execution on mobile platforms.
arXiv Detail & Related papers (2020-01-20T16:17:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.