Shapley Pruning for Neural Network Compression

• URL: http://arxiv.org/abs/2407.15875v1
• Date: Fri, 19 Jul 2024 11:42:54 GMT
• Title: Shapley Pruning for Neural Network Compression
• Authors: Kamil Adamczewski, Yawei Li, Luc van Gool,
• Abstract summary: This work presents the Shapley value approximations, and performs the comparative analysis in terms of cost-benefit utility for the neural network compression. The proposed normative ranking and its approximations show practical results, obtaining state-of-the-art network compression.
• Score: 63.60286036508473
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural network pruning is a rich field with a variety of approaches. In this work, we propose to connect the existing pruning concepts such as leave-one-out pruning and oracle pruning and develop them into a more general Shapley value-based framework that targets the compression of convolutional neural networks. To allow for practical applications in utilizing the Shapley value, this work presents the Shapley value approximations, and performs the comparative analysis in terms of cost-benefit utility for the neural network compression. The proposed ranks are evaluated against a new benchmark, Oracle rank, constructed based on oracle sets. The broad experiments show that the proposed normative ranking and its approximations show practical results, obtaining state-of-the-art network compression.

Related papers

• Sparsest Models Elude Pruning: An Exposé of Pruning's Current Capabilities [4.842973374883628]
  Pruning has emerged as a promising approach for compressing large-scale models, yet its effectiveness in recovering the sparsest of models has not yet been explored. We conducted an extensive series of 485,838 experiments, applying a range of state-of-the-art pruning algorithms to a synthetic dataset we created, named the Cubist Spiral. Our findings reveal a significant gap in performance compared to ideal sparse networks, which we identified through a novel search algorithm.
  arXiv  Detail & Related papers  (2024-07-04T17:33:15Z)
• Fast Shapley Value Estimation: A Unified Approach [71.92014859992263]
  We propose a straightforward and efficient Shapley estimator, SimSHAP, by eliminating redundant techniques. In our analysis of existing approaches, we observe that estimators can be unified as a linear transformation of randomly summed values from feature subsets. Our experiments validate the effectiveness of our SimSHAP, which significantly accelerates the computation of accurate Shapley values.
  arXiv  Detail & Related papers  (2023-11-02T06:09:24Z)
• Robust Training and Verification of Implicit Neural Networks: A Non-Euclidean Contractive Approach [64.23331120621118]
  This paper proposes a theoretical and computational framework for training and robustness verification of implicit neural networks. We introduce a related embedded network and show that the embedded network can be used to provide an $ell_infty$-norm box over-approximation of the reachable sets of the original network. We apply our algorithms to train implicit neural networks on the MNIST dataset and compare the robustness of our models with the models trained via existing approaches in the literature.
  arXiv  Detail & Related papers  (2022-08-08T03:13:24Z)
• i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery [11.119895959906085]
  We propose a novel, structured pruning algorithm for neural networks -- the iterative, Sparse Structured Pruning, dubbed as i-SpaSP. i-SpaSP operates by identifying a larger set of important parameter groups within a network that contribute most to the residual between pruned and dense network output. It is shown to discover high-performing sub-networks and improve upon the pruning efficiency of provable baseline methodologies by several orders of magnitude.
  arXiv  Detail & Related papers  (2021-12-07T05:26:45Z)
• On the Compression of Neural Networks Using $\ell_0$-Norm Regularization and Weight Pruning [0.9821874476902968]
  The present paper is dedicated to the development of a novel compression scheme for neural networks. A new form of regularization is firstly developed, which is capable of inducing strong sparseness in the network during training. The proposed compression scheme also involves the use of $ell$-norm regularization to avoid overfitting as well as fine tuning to improve the performance of the pruned network.
  arXiv  Detail & Related papers  (2021-09-10T19:19:42Z)
• Synthesis and Pruning as a Dynamic Compression Strategy for Efficient Deep Neural Networks [1.8275108630751844]
  We propose a novel strategic synthesis algorithm for feedforward networks that draws directly from the brain's behaviours when learning. Unlike existing approaches that advocate random selection, we select highly performing nodes as starting points for new edges. The strategy aims only to produce useful connections and result in a smaller residual network structure.
  arXiv  Detail & Related papers  (2020-11-23T12:30:57Z)
• ESPN: Extremely Sparse Pruned Networks [50.436905934791035]
  We show that a simple iterative mask discovery method can achieve state-of-the-art compression of very deep networks. Our algorithm represents a hybrid approach between single shot network pruning methods and Lottery-Ticket type approaches.
  arXiv  Detail & Related papers  (2020-06-28T23:09:27Z)
• Shapley Value as Principled Metric for Structured Network Pruning [10.96182578337852]
  Structured pruning is a technique to reduce the storage size and inference cost of neural networks. We show that reducing the harm caused by pruning becomes crucial to retain the performance of the network. We propose Shapley values as a principled ranking metric for this task.
  arXiv  Detail & Related papers  (2020-06-02T17:26:49Z)
• Distance-Based Regularisation of Deep Networks for Fine-Tuning [116.71288796019809]
  We develop an algorithm that constrains a hypothesis class to a small sphere centred on the initial pre-trained weights. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results.
  arXiv  Detail & Related papers  (2020-02-19T16:00:47Z)
• Understanding Generalization in Deep Learning via Tensor Methods [53.808840694241]
  We advance the understanding of the relations between the network's architecture and its generalizability from the compression perspective. We propose a series of intuitive, data-dependent and easily-measurable properties that tightly characterize the compressibility and generalizability of neural networks.
  arXiv  Detail & Related papers  (2020-01-14T22:26:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.