Towards Efficient Deep Spiking Neural Networks Construction with Spiking Activity based Pruning

• URL: http://arxiv.org/abs/2406.01072v1
• Date: Mon, 3 Jun 2024 07:44:37 GMT
• Title: Towards Efficient Deep Spiking Neural Networks Construction with Spiking Activity based Pruning
• Authors: Yaxin Li, Qi Xu, Jiangrong Shen, Hongming Xu, Long Chen, Gang Pan,
• Abstract summary: We propose a structured pruning approach based on the activity levels of convolutional kernels named Spiking Channel Activity-based (SCA) network pruning framework. Inspired by synaptic plasticity mechanisms, our method dynamically adjusts the network's structure by pruning and regenerating convolutional kernels during training, enhancing the model's adaptation to the current target task.
• Score: 17.454100169491497
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The emergence of deep and large-scale spiking neural networks (SNNs) exhibiting high performance across diverse complex datasets has led to a need for compressing network models due to the presence of a significant number of redundant structural units, aiming to more effectively leverage their low-power consumption and biological interpretability advantages. Currently, most model compression techniques for SNNs are based on unstructured pruning of individual connections, which requires specific hardware support. Hence, we propose a structured pruning approach based on the activity levels of convolutional kernels named Spiking Channel Activity-based (SCA) network pruning framework. Inspired by synaptic plasticity mechanisms, our method dynamically adjusts the network's structure by pruning and regenerating convolutional kernels during training, enhancing the model's adaptation to the current target task. While maintaining model performance, this approach refines the network architecture, ultimately reducing computational load and accelerating the inference process. This indicates that structured dynamic sparse learning methods can better facilitate the application of deep SNNs in low-power and high-efficiency scenarios.

Related papers

• Convergence Analysis for Deep Sparse Coding via Convolutional Neural Networks [7.956678963695681]
  We introduce a novel class of Deep Sparse Coding (DSC) models. We derive convergence rates for CNNs in their ability to extract sparse features. Inspired by the strong connection between sparse coding and CNNs, we explore training strategies to encourage neural networks to learn more sparse features.
  arXiv  Detail & Related papers  (2024-08-10T12:43:55Z)
• Self Expanding Convolutional Neural Networks [1.4330085996657045]
  We present a novel method for dynamically expanding Convolutional Neural Networks (CNNs) during training. We employ a strategy where a single model is dynamically expanded, facilitating the extraction of checkpoints at various complexity levels.
  arXiv  Detail & Related papers  (2024-01-11T06:22:40Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• A Faster Approach to Spiking Deep Convolutional Neural Networks [0.0]
  Spiking neural networks (SNNs) have closer dynamics to the brain than current deep neural networks. We propose a network structure based on previous work to improve network runtime and accuracy.
  arXiv  Detail & Related papers  (2022-10-31T16:13:15Z)
• STN: Scalable Tensorizing Networks via Structure-Aware Training and Adaptive Compression [10.067082377396586]
  We propose Scalableizing Networks (STN), which adaptively adjust the model size and decomposition structure without retraining. STN is compatible with arbitrary network architectures and achieves higher compression performance and flexibility over other tensorizing versions.
  arXiv  Detail & Related papers  (2022-05-30T15:50:48Z)
• Self-Reorganizing and Rejuvenating CNNs for Increasing Model Capacity Utilization [8.661269034961679]
  We propose a biologically inspired method for improving the computational resource utilization of neural networks. The proposed method utilizes the channel activations of a convolution layer in order to reorganize that layers parameters. The rejuvenated parameters learn different features to supplement those learned by the reorganized surviving parameters.
  arXiv  Detail & Related papers  (2021-02-13T06:19:45Z)
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search [55.164053971213576]
  convolutional neural network has achieved great success in fulfilling computer vision tasks despite large computation overhead. Structured (channel) pruning is usually applied to reduce the model redundancy while preserving the network structure. Existing structured pruning methods require hand-crafted rules which may lead to tremendous pruning space.
  arXiv  Detail & Related papers  (2020-11-04T07:43:01Z)
• Structured Convolutions for Efficient Neural Network Design [65.36569572213027]
  We tackle model efficiency by exploiting redundancy in the textitimplicit structure of the building blocks of convolutional neural networks. We show how this decomposition can be applied to 2D and 3D kernels as well as the fully-connected layers.
  arXiv  Detail & Related papers  (2020-08-06T04:38:38Z)
• An Ode to an ODE [78.97367880223254]
  We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the group O(d) This nested system of two flows provides stability and effectiveness of training and provably solves the gradient vanishing-explosion problem.
  arXiv  Detail & Related papers  (2020-06-19T22:05:19Z)
• Structured Sparsification with Joint Optimization of Group Convolution and Channel Shuffle [117.95823660228537]
  We propose a novel structured sparsification method for efficient network compression. The proposed method automatically induces structured sparsity on the convolutional weights. We also address the problem of inter-group communication with a learnable channel shuffle mechanism.
  arXiv  Detail & Related papers  (2020-02-19T12:03:10Z)
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment [84.57874289554839]
  Training deep neural networks on large-scale datasets requires significant hardware resources. Backpropagation, the workhorse for training these networks, is an inherently sequential process that is difficult to parallelize. We propose a neuro-biologically-plausible alternative to backprop that can be used to train deep networks.
  arXiv  Detail & Related papers  (2020-02-10T16:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.