A Probabilistic Approach to Neural Network Pruning

• URL: http://arxiv.org/abs/2105.10065v1
• Date: Thu, 20 May 2021 23:19:43 GMT
• Title: A Probabilistic Approach to Neural Network Pruning
• Authors: Xin Qian, Diego Klabjan
• Abstract summary: We theoretically study the performance of two pruning techniques (random and magnitude-based) on FCNs and CNNs. The results establish that there exist pruned networks with expressive power within any specified bound from the target network.
• Score: 20.001091112545065
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural network pruning techniques reduce the number of parameters without compromising predicting ability of a network. Many algorithms have been developed for pruning both over-parameterized fully-connected networks (FCNs) and convolutional neural networks (CNNs), but analytical studies of capabilities and compression ratios of such pruned sub-networks are lacking. We theoretically study the performance of two pruning techniques (random and magnitude-based) on FCNs and CNNs. Given a target network {whose weights are independently sampled from appropriate distributions}, we provide a universal approach to bound the gap between a pruned and the target network in a probabilistic sense. The results establish that there exist pruned networks with expressive power within any specified bound from the target network.

Related papers

• Using Cooperative Game Theory to Prune Neural Networks [7.3959659158152355]
  We show how solution concepts from cooperative game theory can be used to tackle the problem of pruning neural networks. We introduce a method called Game Theory Assisted Pruning (GTAP), which reduces the neural network's size while preserving its predictive accuracy.
  arXiv  Detail & Related papers  (2023-11-17T11:48:10Z)
• Certified Invertibility in Neural Networks via Mixed-Integer Programming [16.64960701212292]
  Neural networks are known to be vulnerable to adversarial attacks. There may exist large, meaningful perturbations that do not affect the network's decision. We discuss how our findings can be useful for invertibility certification in transformations between neural networks.
  arXiv  Detail & Related papers  (2023-01-27T15:40:38Z)
• Probabilistic Verification of ReLU Neural Networks via Characteristic Functions [11.489187712465325]
  We use ideas from probability theory in the frequency domain to provide probabilistic verification guarantees for ReLU neural networks. We interpret a (deep) feedforward neural network as a discrete dynamical system over a finite horizon. We obtain the corresponding cumulative distribution function of the output set, which can be used to check if the network is performing as expected.
  arXiv  Detail & Related papers  (2022-12-03T05:53:57Z)
• Robust Training and Verification of Implicit Neural Networks: A Non-Euclidean Contractive Approach [64.23331120621118]
  This paper proposes a theoretical and computational framework for training and robustness verification of implicit neural networks. We introduce a related embedded network and show that the embedded network can be used to provide an $ell_infty$-norm box over-approximation of the reachable sets of the original network. We apply our algorithms to train implicit neural networks on the MNIST dataset and compare the robustness of our models with the models trained via existing approaches in the literature.
  arXiv  Detail & Related papers  (2022-08-08T03:13:24Z)
• Variational Neural Networks [88.24021148516319]
  We propose a method for uncertainty estimation in neural networks called Variational Neural Network (VNN) VNN generates parameters for the output distribution of a layer by transforming its inputs with learnable sub-layers. In uncertainty quality estimation experiments, we show that VNNs achieve better uncertainty quality than Monte Carlo Dropout or Bayes By Backpropagation methods.
  arXiv  Detail & Related papers  (2022-07-04T15:41:02Z)
• On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks [91.3755431537592]
  We study how random pruning of the weights affects a neural network's neural kernel (NTK) In particular, this work establishes an equivalence of the NTKs between a fully-connected neural network and its randomly pruned version.
  arXiv  Detail & Related papers  (2022-03-27T15:22:19Z)
• Finding Everything within Random Binary Networks [11.689913953698081]
  We prove that a random network can be approximated up to arbitrary accuracy by simply pruning a random network of binary $pm1$ weights. We prove that any target network can be approximated up to arbitrary accuracy by simply pruning a random network of binary $pm1$ weights that is only a polylogarithmic factor wider and deeper than the target network.
  arXiv  Detail & Related papers  (2021-10-18T03:19:25Z)
• Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification [10.727102755903616]
  We aim for efficient deep BNNs amenable to complex computer vision architectures. We achieve this by leveraging variational autoencoders (VAEs) to learn the interaction and the latent distribution of the parameters at each network layer. Our approach, Latent-Posterior BNN (LP-BNN), is compatible with the recent BatchEnsemble method, leading to highly efficient (in terms of computation and memory during both training and testing) ensembles.
  arXiv  Detail & Related papers  (2020-12-04T19:50:09Z)
• Finite Versus Infinite Neural Networks: an Empirical Study [69.07049353209463]
  kernel methods outperform fully-connected finite-width networks. Centered and ensembled finite networks have reduced posterior variance. Weight decay and the use of a large learning rate break the correspondence between finite and infinite networks.
  arXiv  Detail & Related papers  (2020-07-31T01:57:47Z)
• ESPN: Extremely Sparse Pruned Networks [50.436905934791035]
  We show that a simple iterative mask discovery method can achieve state-of-the-art compression of very deep networks. Our algorithm represents a hybrid approach between single shot network pruning methods and Lottery-Ticket type approaches.
  arXiv  Detail & Related papers  (2020-06-28T23:09:27Z)
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio [101.84651388520584]
  This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs. Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-06T15:51:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.