The fine line between dead neurons and sparsity in binarized spiking neural networks

• URL: http://arxiv.org/abs/2201.11915v1
• Date: Fri, 28 Jan 2022 03:33:12 GMT
• Title: The fine line between dead neurons and sparsity in binarized spiking neural networks
• Authors: Jason K. Eshraghian, Wei D. Lu
• Abstract summary: Spiking neural networks can compensate for quantization error by encoding information or processing discretized quantities. In this paper, we propose the use of threshold annealing' as a warm-up method for firing thresholds. We show it enables the propagation of spikes across multiple layers where neurons would otherwise cease to fire, and in doing so, achieve highly competitive results on four diverse datasets.
• Score: 1.370633147306388
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spiking neural networks can compensate for quantization error by encoding information either in the temporal domain, or by processing discretized quantities in hidden states of higher precision. In theory, a wide dynamic range state-space enables multiple binarized inputs to be accumulated together, thus improving the representational capacity of individual neurons. This may be achieved by increasing the firing threshold, but make it too high and sparse spike activity turns into no spike emission. In this paper, we propose the use of `threshold annealing' as a warm-up method for firing thresholds. We show it enables the propagation of spikes across multiple layers where neurons would otherwise cease to fire, and in doing so, achieve highly competitive results on four diverse datasets, despite using binarized weights. Source code is available at https://github.com/jeshraghian/snn-tha/

Related papers

• Fully Spiking Actor Network with Intra-layer Connections for Reinforcement Learning [51.386945803485084]
  We focus on the task where the agent needs to learn multi-dimensional deterministic policies to control. Most existing spike-based RL methods take the firing rate as the output of SNNs, and convert it to represent continuous action space (i.e., the deterministic policy) through a fully-connected layer. To develop a fully spiking actor network without any floating-point matrix operations, we draw inspiration from the non-spiking interneurons found in insects.
  arXiv  Detail & Related papers  (2024-01-09T07:31:34Z)
• Ternary Spike: Learning Ternary Spikes for Spiking Neural Networks [19.304952813634994]
  Spiking Neural Network (SNN) is a biologically inspired neural network infrastructures. In this paper, we propose a ternary spike neuron to transmit information. We show that the ternary spike can consistently outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-11T13:28:54Z)
• Variable Bitrate Neural Fields [75.24672452527795]
  We present a dictionary method for compressing feature grids, reducing their memory consumption by up to 100x. We formulate the dictionary optimization as a vector-quantized auto-decoder problem which lets us learn end-to-end discrete neural representations in a space where no direct supervision is available.
  arXiv  Detail & Related papers  (2022-06-15T17:58:34Z)
• Astrocytes mediate analogous memory in a multi-layer neuron-astrocytic network [52.77024349608834]
  We show how a piece of information can be maintained as a robust activity pattern for several seconds then completely disappear if no other stimuli come. This kind of short-term memory can keep operative information for seconds, then completely forget it to avoid overlapping with forthcoming patterns. We show how arbitrary patterns can be loaded, then stored for a certain interval of time, and retrieved if the appropriate clue pattern is applied to the input.
  arXiv  Detail & Related papers  (2021-08-31T16:13:15Z)
• Hessian Aware Quantization of Spiking Neural Networks [1.90365714903665]
  Neuromorphic architecture allows massively parallel computation with variable and local bit-precisions. Current gradient based methods of SNN training use a complex neuron model with multiple state variables. We present a simplified neuron model that reduces the number of state variables by 4-fold while still being compatible with gradient based training.
  arXiv  Detail & Related papers  (2021-04-29T05:27:34Z)
• Efficient bit encoding of neural networks for Fock states [77.34726150561087]
  The complexity of the neural network scales only with the number of bit-encoded neurons rather than the maximum boson number. In the high occupation regime its information compression efficiency is shown to surpass even maximally optimized density matrix implementations.
  arXiv  Detail & Related papers  (2021-03-15T11:24:40Z)
• Artificial Neural Networks generated by Low Discrepancy Sequences [59.51653996175648]
  We generate artificial neural networks as random walks on a dense network graph. Such networks can be trained sparse from scratch, avoiding the expensive procedure of training a dense network and compressing it afterwards. We demonstrate that the artificial neural networks generated by low discrepancy sequences can achieve an accuracy within reach of their dense counterparts at a much lower computational complexity.
  arXiv  Detail & Related papers  (2021-03-05T08:45:43Z)
• Neuron Merging: Compensating for Pruned Neurons [3.441021278275805]
  Structured network pruning discards the whole neuron or filter, leading to accuracy loss. We propose a novel concept of neuron merging applicable to both fully connected layers and convolution layers. We achieve an accuracy of 93.16% while reducing 64% of total parameters, without any fine-tuning.
  arXiv  Detail & Related papers  (2020-10-25T16:50:26Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)
• Optimized spiking neurons classify images with high accuracy through temporal coding with two spikes [1.7767466724342065]
  Spike-based neuromorphic hardware promises to reduce the energy consumption of image classification and other deep learning applications. Previous methods for converting trained artificial neural networks to spiking neurons were inefficient because the neurons had to emit too many spikes. We show that a substantially more efficient conversion arises when one optimize the spiking neuron model for that purpose.
  arXiv  Detail & Related papers  (2020-01-31T10:11:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.