Forward Direct Feedback Alignment for Online Gradient Estimates of Spiking Neural Networks

• URL: http://arxiv.org/abs/2403.08804v1
• Date: Tue, 6 Feb 2024 09:07:12 GMT
• Title: Forward Direct Feedback Alignment for Online Gradient Estimates of Spiking Neural Networks
• Authors: Florian Bacho, Dminique Chu,
• Abstract summary: Spiking neural networks can be simulated energy efficiently on neuromorphic hardware platforms. We propose a novel neuromorphic algorithm, the textitSpiking Forward Direct Feedback Alignment (SFDFA) algorithm.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: There is an interest in finding energy efficient alternatives to current state of the art neural network training algorithms. Spiking neural network are a promising approach, because they can be simulated energy efficiently on neuromorphic hardware platforms. However, these platforms come with limitations on the design of the training algorithm. Most importantly, backpropagation cannot be implemented on those. We propose a novel neuromorphic algorithm, the \textit{Spiking Forward Direct Feedback Alignment} (SFDFA) algorithm, an adaption of \textit{Forward Direct Feedback Alignment} to train SNNs. SFDFA estimates the weights between output and hidden neurons as feedback connections. The main contribution of this paper is to describe how exact local gradients of spikes can be computed in an online manner while taking into account the intra-neuron dependencies between post-synaptic spikes and derive a dynamical system for neuromorphic hardware compatibility. We compare the SFDFA algorithm with a number of competitor algorithms and show that the proposed algorithm achieves higher performance and convergence rates.

Related papers

• Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks [69.2642802272367]
  Brain-inspired neuromorphic computing with spiking neural networks (SNNs) is a promising energy-efficient computational approach. Most recent methods leverage spatial and temporal backpropagation (BP), not adhering to neuromorphic properties. We propose a novel method, online pseudo-zeroth-order (OPZO) training.
  arXiv  Detail & Related papers  (2024-07-17T12:09:00Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• The Cascaded Forward Algorithm for Neural Network Training [61.06444586991505]
  We propose a new learning framework for neural networks, namely Cascaded Forward (CaFo) algorithm, which does not rely on BP optimization as that in FF. Unlike FF, our framework directly outputs label distributions at each cascaded block, which does not require generation of additional negative samples. In our framework each block can be trained independently, so it can be easily deployed into parallel acceleration systems.
  arXiv  Detail & Related papers  (2023-03-17T02:01:11Z)
• Permutation Equivariant Neural Functionals [92.0667671999604]
  This work studies the design of neural networks that can process the weights or gradients of other neural networks. We focus on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order. In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks.
  arXiv  Detail & Related papers  (2023-02-27T18:52:38Z)
• Exact Gradient Computation for Spiking Neural Networks Through Forward Propagation [39.33537954568678]
  Spiking neural networks (SNN) have emerged as alternatives to traditional neural networks. We propose a novel training algorithm, called emphforward propagation (FP), that computes exact gradients for SNN.
  arXiv  Detail & Related papers  (2022-10-18T20:28:21Z)
• FFNB: Forgetting-Free Neural Blocks for Deep Continual Visual Learning [14.924672048447338]
  We devise a dynamic network architecture for continual learning based on a novel forgetting-free neural block (FFNB) Training FFNB features on new tasks is achieved using a novel procedure that constrains the underlying parameters in the null-space of the previous tasks.
  arXiv  Detail & Related papers  (2021-11-22T17:23:34Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Online Limited Memory Neural-Linear Bandits with Likelihood Matching [53.18698496031658]
  We study neural-linear bandits for solving problems where both exploration and representation learning play an important role. We propose a likelihood matching algorithm that is resilient to catastrophic forgetting and is completely online.
  arXiv  Detail & Related papers  (2021-02-07T14:19:07Z)
• Brain-Inspired Learning on Neuromorphic Substrates [5.279475826661643]
  This article provides a mathematical framework for the design of practical online learning algorithms for neuromorphic substrates. Specifically, we show a direct connection between Real-Time Recurrent Learning (RTRL) and biologically plausible learning rules for training Spiking Neural Networks (SNNs) We motivate a sparse approximation based on block-diagonal Jacobians, which reduces the algorithm's computational complexity.
  arXiv  Detail & Related papers  (2020-10-22T17:56:59Z)
• Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment [4.280642750854163]
  Training neural networks for neuromorphic deployment is non-trivial. We introduce a Bayesian approach for optimizing the hyper parameters of an algorithm for training binary communication networks that can be deployed to neuromorphic hardware. We show that by optimizing the hyper parameters on this algorithm for each dataset, we can achieve improvements in accuracy over the previous state-of-the-art for this algorithm on each dataset.
  arXiv  Detail & Related papers  (2020-04-21T01:15:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.