Emerging NeoHebbian Dynamics in Forward-Forward Learning: Implications for Neuromorphic Computing

• URL: http://arxiv.org/abs/2406.16479v1
• Date: Mon, 24 Jun 2024 09:33:56 GMT
• Title: Emerging NeoHebbian Dynamics in Forward-Forward Learning: Implications for Neuromorphic Computing
• Authors: Erik B. Terres-Escudero, Javier Del Ser, Pablo García-Bringas,
• Abstract summary: Forward-Forward Algorithm (FFA) employs local learning rules for each layer. We show that when employing a squared Euclidean norm as a goodness function driving the local learning, the resulting FFA is equivalent to a neo-Hebbian Learning Rule.
• Score: 7.345136916791223
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Advances in neural computation have predominantly relied on the gradient backpropagation algorithm (BP). However, the recent shift towards non-stationary data modeling has highlighted the limitations of this heuristic, exposing that its adaptation capabilities are far from those seen in biological brains. Unlike BP, where weight updates are computed through a reverse error propagation path, Hebbian learning dynamics provide synaptic updates using only information within the layer itself. This has spurred interest in biologically plausible learning algorithms, hypothesized to overcome BP's shortcomings. In this context, Hinton recently introduced the Forward-Forward Algorithm (FFA), which employs local learning rules for each layer and has empirically proven its efficacy in multiple data modeling tasks. In this work we argue that when employing a squared Euclidean norm as a goodness function driving the local learning, the resulting FFA is equivalent to a neo-Hebbian Learning Rule. To verify this result, we compare the training behavior of FFA in analog networks with its Hebbian adaptation in spiking neural networks. Our experiments demonstrate that both versions of FFA produce similar accuracy and latent distributions. The findings herein reported provide empirical evidence linking biological learning rules with currently used training algorithms, thus paving the way towards extrapolating the positive outcomes from FFA to Hebbian learning rules. Simultaneously, our results imply that analog networks trained under FFA could be directly applied to neuromorphic computing, leading to reduced energy usage and increased computational speed.

Related papers

• A Contrastive Symmetric Forward-Forward Algorithm (SFFA) for Continual Learning Tasks [7.345136916791223]
  Forward-Forward Algorithm (FFA) has recently gained momentum as an alternative to the conventional back-propagation algorithm for neural network learning. This work proposes the Symmetric Forward-Forward Algorithm (SFFA), a novel modification of the original FFA which partitions each layer into positive and negative neurons.
  arXiv  Detail & Related papers  (2024-09-11T16:21:44Z)
• On the Improvement of Generalization and Stability of Forward-Only Learning via Neural Polarization [7.345136916791223]
  Forward-Forward Algorithm (FFA) has been shown to achieve competitive levels of performance in terms of generalization and complexity. We propose a novel implementation of the FFA algorithm, denoted as Polar-FFA, which extends the original formulation by introducing a neural division. Our results demonstrate that Polar-FFA outperforms FFA in terms of accuracy and convergence speed.
  arXiv  Detail & Related papers  (2024-08-17T14:32:18Z)
• SGD method for entropy error function with smoothing l0 regularization for neural networks [3.108634881604788]
  entropy error function has been widely used in neural networks. We propose a novel entropy function with smoothing l0 regularization for feed-forward neural networks. Our work is novel as it enables neural networks to learn effectively, producing more accurate predictions.
  arXiv  Detail & Related papers  (2024-05-28T19:54:26Z)
• The Cascaded Forward Algorithm for Neural Network Training [61.06444586991505]
  We propose a new learning framework for neural networks, namely Cascaded Forward (CaFo) algorithm, which does not rely on BP optimization as that in FF. Unlike FF, our framework directly outputs label distributions at each cascaded block, which does not require generation of additional negative samples. In our framework each block can be trained independently, so it can be easily deployed into parallel acceleration systems.
  arXiv  Detail & Related papers  (2023-03-17T02:01:11Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Convolutional generative adversarial imputation networks for spatio-temporal missing data in storm surge simulations [86.5302150777089]
  Generative Adversarial Imputation Nets (GANs) and GAN-based techniques have attracted attention as unsupervised machine learning methods. We name our proposed method as Con Conval Generative Adversarial Imputation Nets (Conv-GAIN)
  arXiv  Detail & Related papers  (2021-11-03T03:50:48Z)
• FF-NSL: Feed-Forward Neural-Symbolic Learner [70.978007919101]
  This paper introduces a neural-symbolic learning framework, called Feed-Forward Neural-Symbolic Learner (FF-NSL) FF-NSL integrates state-of-the-art ILP systems based on the Answer Set semantics, with neural networks, in order to learn interpretable hypotheses from labelled unstructured data.
  arXiv  Detail & Related papers  (2021-06-24T15:38:34Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Predictive Coding Can Do Exact Backpropagation on Any Neural Network [40.51949948934705]
  We generalize (IL and) Z-IL by directly defining them on computational graphs. This is the first biologically plausible algorithm that is shown to be equivalent to BP in the way of updating parameters on any neural network.
  arXiv  Detail & Related papers  (2021-03-08T11:52:51Z)
• MAP Propagation Algorithm: Faster Learning with a Team of Reinforcement Learning Agents [0.0]
  An alternative way of training an artificial neural network is through treating each unit in the network as a reinforcement learning agent. We propose a novel algorithm called MAP propagation to reduce this variance significantly. Our work thus allows for the broader application of the teams of agents in deep reinforcement learning.
  arXiv  Detail & Related papers  (2020-10-15T17:17:39Z)
• Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks [55.0627904986664]
  Spiking Neural Networks (SNNs) usetemporal spike patterns to represent and transmit information, which is not only biologically realistic but also suitable for ultra-low-power event-driven neuromorphic implementation. This paper investigates the contribution of spike timing dynamics to information encoding, synaptic plasticity and decision making, providing a new perspective to design of future DeepSNNs and neuromorphic hardware systems.
  arXiv  Detail & Related papers  (2020-03-26T11:13:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.