SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance

• URL: http://arxiv.org/abs/2207.04089v1
• Date: Fri, 8 Jul 2022 18:27:42 GMT
• Title: SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance
• Authors: Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, Kevin Bailly
• Abstract summary: We propose a novel integrated gradient pruning criterion, in which the relevance of each neuron is defined as the integral of the gradient variation on a path towards this neuron removal. We show through extensive validation on several datasets, architectures as well as pruning scenarios that the proposed method, dubbed SInGE, significantly outperforms existing state-of-the-art pruning methods.
• Score: 37.82255888371488
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The leap in performance in state-of-the-art computer vision methods is attributed to the development of deep neural networks. However it often comes at a computational price which may hinder their deployment. To alleviate this limitation, structured pruning is a well known technique which consists in removing channels, neurons or filters, and is commonly applied in order to produce more compact models. In most cases, the computations to remove are selected based on a relative importance criterion. At the same time, the need for explainable predictive models has risen tremendously and motivated the development of robust attribution methods that highlight the relative importance of pixels of an input image or feature map. In this work, we discuss the limitations of existing pruning heuristics, among which magnitude and gradient-based methods. We draw inspiration from attribution methods to design a novel integrated gradient pruning criterion, in which the relevance of each neuron is defined as the integral of the gradient variation on a path towards this neuron removal. Furthermore, we propose an entwined DNN pruning and fine-tuning flowchart to better preserve DNN accuracy while removing parameters. We show through extensive validation on several datasets, architectures as well as pruning scenarios that the proposed method, dubbed SInGE, significantly outperforms existing state-of-the-art DNN pruning methods.

Related papers

• Spatiotemporal Learning on Cell-embedded Graphs [6.8090864965073274]
  We introduce a learnable cell attribution to the node-edge message passing process, which better captures the spatial dependency of regional features. Experiments on various PDE systems and one real-world dataset demonstrate that CeGNN achieves superior performance compared with other baseline models.
  arXiv  Detail & Related papers  (2024-09-26T16:22:08Z)
• Geometric sparsification in recurrent neural networks [0.8851237804522972]
  We propose a new technique for sparsification of recurrent neural nets (RNNs) called moduli regularization. We show that moduli regularization induces more stable RNNs with a variety of moduli regularizers, and achieves high fidelity models at 98% sparsity.
  arXiv  Detail & Related papers  (2024-06-10T14:12:33Z)
• Model-Based Control with Sparse Neural Dynamics [23.961218902837807]
  We propose a new framework for integrated model learning and predictive control. We show that our framework can deliver better closed-loop performance than existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-20T06:25:02Z)
• Inducing Gaussian Process Networks [80.40892394020797]
  We propose inducing Gaussian process networks (IGN), a simple framework for simultaneously learning the feature space as well as the inducing points. The inducing points, in particular, are learned directly in the feature space, enabling a seamless representation of complex structured domains. We report on experimental results for real-world data sets showing that IGNs provide significant advances over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-04-21T05:27:09Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Non-Gradient Manifold Neural Network [79.44066256794187]
  Deep neural network (DNN) generally takes thousands of iterations to optimize via gradient descent. We propose a novel manifold neural network based on non-gradient optimization.
  arXiv  Detail & Related papers  (2021-06-15T06:39:13Z)
• A novel Deep Neural Network architecture for non-linear system identification [78.69776924618505]
  We present a novel Deep Neural Network (DNN) architecture for non-linear system identification. Inspired by fading memory systems, we introduce inductive bias (on the architecture) and regularization (on the loss function) This architecture allows for automatic complexity selection based solely on available data.
  arXiv  Detail & Related papers  (2021-06-06T10:06:07Z)
• A Differentiable Point Process with Its Application to Spiking Neural Networks [13.160616423673373]
  Jimenez Rezende & Gerstner (2014) proposed a variational inference algorithm to train SNNs with hidden neurons. This paper presents an alternative gradient estimator for SNNs based on the path-wise gradient estimator.
  arXiv  Detail & Related papers  (2021-06-02T02:40:17Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.