What is the State of Neural Network Pruning?

• URL: http://arxiv.org/abs/2003.03033v1
• Date: Fri, 6 Mar 2020 05:06:12 GMT
• Title: What is the State of Neural Network Pruning?
• Authors: Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, John Guttag
• Abstract summary: We provide a meta-analysis of the literature, including an overview of approaches to pruning. We find that the community suffers from a lack of standardized benchmarks and metrics. We introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods.
• Score: 12.50128492336137
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural network pruning---the task of reducing the size of a network by removing parameters---has been the subject of a great deal of work in recent years. We provide a meta-analysis of the literature, including an overview of approaches to pruning and consistent findings in the literature. After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics. This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades. To address this situation, we identify issues with current practices, suggest concrete remedies, and introduce ShrinkBench, an open-source framework to facilitate standardized evaluations of pruning methods. We use ShrinkBench to compare various pruning techniques and show that its comprehensive evaluation can prevent common pitfalls when comparing pruning methods.

Related papers

• Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness [16.623648447423438]
  Recent work has proposed neural network pruning techniques to reduce the size of a network while preserving robustness against adversarial examples. These methods involve complex and articulated designs, making it difficult to analyze the differences and establish a fair and accurate comparison. We propose a novel taxonomy to categorize them based on two main dimensions: the pipeline, defining when to prune; and the specifics, defining how to prune.
  arXiv  Detail & Related papers  (2024-09-02T13:34:01Z)
• Towards Evaluating Transfer-based Attacks Systematically, Practically, and Fairly [79.07074710460012]
  adversarial vulnerability of deep neural networks (DNNs) has drawn great attention. An increasing number of transfer-based methods have been developed to fool black-box DNN models. We establish a transfer-based attack benchmark (TA-Bench) which implements 30+ methods.
  arXiv  Detail & Related papers  (2023-11-02T15:35:58Z)
• ThinResNet: A New Baseline for Structured Convolutional Networks Pruning [1.90298817989995]
  Pruning is a compression method which aims to improve the efficiency of neural networks by reducing their number of parameters. In this work, we verify how results in the recent literature of pruning hold up against networks that underwent both state-of-the-art training methods and trivial model scaling.
  arXiv  Detail & Related papers  (2023-09-22T13:28:18Z)
• Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruning [58.34310957892895]
  The state of neural network pruning has been noticed to be unclear and even confusing for a while. We first clarify the fairness principle in pruning experiments and summarize the widely-used comparison setups. We then point out the central role of network trainability, which has not been well recognized so far.
  arXiv  Detail & Related papers  (2023-01-12T18:58:33Z)
• MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models [78.45898846056303]
  Pruning is an effective method to reduce the memory footprint and computational cost associated with large natural language processing models. We develop a novel MultiLevel structured Pruning framework, which uses three different levels of structured pruning: head pruning, row pruning, and block-wise sparse pruning.
  arXiv  Detail & Related papers  (2021-05-30T22:00:44Z)
• Emerging Paradigms of Neural Network Pruning [82.9322109208353]
  Pruning is adopted as a post-processing solution to this problem, which aims to remove unnecessary parameters in a neural network with little performance compromised. Recent works challenge this belief by discovering random sparse networks which can be trained to match the performance with their dense counterpart. This survey seeks to bridge the gap by proposing a general pruning framework so that the emerging pruning paradigms can be accommodated well with the traditional one.
  arXiv  Detail & Related papers  (2021-03-11T05:01:52Z)
• Manifold Regularized Dynamic Network Pruning [102.24146031250034]
  This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
  arXiv  Detail & Related papers  (2021-03-10T03:59:03Z)
• Neural Pruning via Growing Regularization [82.9322109208353]
  We extend regularization to tackle two central problems of pruning: pruning schedule and weight importance scoring. Specifically, we propose an L2 regularization variant with rising penalty factors and show it can bring significant accuracy gains. The proposed algorithms are easy to implement and scalable to large datasets and networks in both structured and unstructured pruning.
  arXiv  Detail & Related papers  (2020-12-16T20:16:28Z)
• Methods for Pruning Deep Neural Networks [0.0]
  This paper presents a survey of methods for pruning deep neural networks. It begins by categorising over 150 studies based on the underlying approach used. It focuses on three categories: methods that use magnitude based pruning, methods that utilise clustering to identify redundancy, and methods that use sensitivity analysis.
  arXiv  Detail & Related papers  (2020-10-31T10:58:51Z)
• Composition of Saliency Metrics for Channel Pruning with a Myopic Oracle [0.8043754868448141]
  Pruning is guided by a pruning saliency, which approximates the change in the loss function associated with the removal of specific weights. Many pruning signals have been proposed, but the performance of each depends on the particular trained network. We propose a method to compose several primitive pruning saliencies, to exploit the cases where each saliency measure does well.
  arXiv  Detail & Related papers  (2020-04-03T11:29:41Z)
• Modeling of Pruning Techniques for Deep Neural Networks Simplification [8.75217589103206]
  Convolutional Neural Networks (CNNs) suffer from different issues, such as computational complexity and the number of parameters. In recent years pruning techniques are employed to reduce the number of operations and model size in CNNs. Various techniques and tricks accompany pruning methods, and there is not a unifying framework to model all the pruning methods.
  arXiv  Detail & Related papers  (2020-01-13T04:51:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.