Scaling SNNs Trained Using Equilibrium Propagation to Convolutional Architectures

• URL: http://arxiv.org/abs/2405.02546v3
• Date: Tue, 2 Jul 2024 20:24:29 GMT
• Title: Scaling SNNs Trained Using Equilibrium Propagation to Convolutional Architectures
• Authors: Jiaqi Lin, Malyaban Bal, Abhronil Sengupta,
• Abstract summary: Equilibrium Propagation (EP) is a biologically plausible local learning algorithm initially developed for convergent recurrent neural networks (RNNs) EP is a powerful candidate for training Spiking Neural Networks (SNNs), which are commonly trained by BPTT. We provide a formulation for training convolutional spiking convergent RNNs using EP, bridging the gap between spiking and non-spiking convergent RNNs.
• Score: 2.2146860305758485
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Equilibrium Propagation (EP) is a biologically plausible local learning algorithm initially developed for convergent recurrent neural networks (RNNs), where weight updates rely solely on the connecting neuron states across two phases. The gradient calculations in EP have been shown to approximate the gradients computed by Backpropagation Through Time (BPTT) when an infinitesimally small nudge factor is used. This property makes EP a powerful candidate for training Spiking Neural Networks (SNNs), which are commonly trained by BPTT. However, in the spiking domain, previous studies on EP have been limited to architectures involving few linear layers. In this work, for the first time we provide a formulation for training convolutional spiking convergent RNNs using EP, bridging the gap between spiking and non-spiking convergent RNNs. We demonstrate that for spiking convergent RNNs, there is a mismatch in the maximum pooling and its inverse operation, leading to inaccurate gradient estimation in EP. Substituting this with average pooling resolves this issue and enables accurate gradient estimation for spiking convergent RNNs. We also highlight the memory efficiency of EP compared to BPTT. In the regime of SNNs trained by EP, our experimental results indicate state-of-the-art performance on the MNIST and FashionMNIST datasets, with test errors of 0.97% and 8.89%, respectively. These results are comparable to those of convergent RNNs and SNNs trained by BPTT. These findings underscore EP as an optimal choice for on-chip training and a biologically-plausible method for computing error gradients.

Related papers

• Speed Limits for Deep Learning [67.69149326107103]
  Recent advancement in thermodynamics allows bounding the speed at which one can go from the initial weight distribution to the final distribution of the fully trained network. We provide analytical expressions for these speed limits for linear and linearizable neural networks. Remarkably, given some plausible scaling assumptions on the NTK spectra and spectral decomposition of the labels -- learning is optimal in a scaling sense.
  arXiv  Detail & Related papers  (2023-07-27T06:59:46Z)
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks [56.35403810762512]
  Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware. We study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method.
  arXiv  Detail & Related papers  (2023-02-01T04:22:59Z)
• Online Training Through Time for Spiking Neural Networks [66.7744060103562]
  Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. We propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning.
  arXiv  Detail & Related papers  (2022-10-09T07:47:56Z)
• Enforcing Continuous Physical Symmetries in Deep Learning Network for Solving Partial Differential Equations [3.6317085868198467]
  We introduce a new method, symmetry-enhanced physics informed neural network (SPINN) where the invariant surface conditions induced by the Lie symmetries of PDEs are embedded into the loss function of PINN. We show that SPINN performs better than PINN with fewer training points and simpler architecture of neural network.
  arXiv  Detail & Related papers  (2022-06-19T00:44:22Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• BioGrad: Biologically Plausible Gradient-Based Learning for Spiking Neural Networks [0.0]
  Spiking neural networks (SNN) are delivering energy-efficient, massively parallel, and low-latency solutions to AI problems. To harness these computational benefits, SNN need to be trained by learning algorithms that adhere to brain-inspired neuromorphic principles. We propose a biologically plausible gradient-based learning algorithm for SNN that is functionally equivalent to backprop.
  arXiv  Detail & Related papers  (2021-10-27T00:07:25Z)
• Spike time displacement based error backpropagation in convolutional spiking neural networks [0.6193838300896449]
  In this paper, we extend the STiDi-BP algorithm to employ it in deeper and convolutional architectures. The evaluation results on the image classification task based on two popular benchmarks, MNIST and Fashion-MNIST, confirm that this algorithm has been applicable in deep SNNs. We consider a convolutional SNN with two sets of weights: real-valued weights that are updated in the backward pass and their signs, binary weights, that are employed in the feedforward process.
  arXiv  Detail & Related papers  (2021-08-31T05:18:59Z)
• Tuning Convolutional Spiking Neural Network with Biologically-plausible Reward Propagation [13.037452551907657]
  Spiking Neural Networks (SNNs) contain more biologically realistic structures and biologically-inspired learning principles. BRP algorithm is proposed and applied to the SNN architecture with both spiking-convolution and full-connection layers.
  arXiv  Detail & Related papers  (2020-10-09T08:42:13Z)
• Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias [65.13042449121411]
  In practice, training a network with the gradient estimates provided by EP does not scale to visual tasks harder than MNIST. We show that a bias in the gradient estimate of EP, inherent in the use of finite nudging, is responsible for this phenomenon. We apply these techniques to train an architecture with asymmetric forward and backward connections, yielding a 13.2% test error.
  arXiv  Detail & Related papers  (2020-06-06T09:36:07Z)
• Continual Weight Updates and Convolutional Architectures for Equilibrium Propagation [69.87491240509485]
  Equilibrium Propagation (EP) is a biologically inspired alternative algorithm to backpropagation (BP) for training neural networks. We propose a discrete-time formulation of EP which enables to simplify equations, speed up training and extend EP to CNNs. Our CNN model achieves the best performance ever reported on MNIST with EP.
  arXiv  Detail & Related papers  (2020-04-29T12:14:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.