GAIT-prop: A biologically plausible learning rule derived from backpropagation of error

• URL: http://arxiv.org/abs/2006.06438v3
• Date: Thu, 5 Nov 2020 18:07:34 GMT
• Title: GAIT-prop: A biologically plausible learning rule derived from backpropagation of error
• Authors: Nasir Ahmad, Marcel A. J. van Gerven, Luca Ambrogioni
• Abstract summary: We show an exact correspondence between backpropagation and a modified form of target propagation. In a series of simple computer vision experiments, we show near-identical performance between backpropagation and GAIT-prop.
• Score: 9.948484577581796
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional backpropagation of error, though a highly successful algorithm for learning in artificial neural network models, includes features which are biologically implausible for learning in real neural circuits. An alternative called target propagation proposes to solve this implausibility by using a top-down model of neural activity to convert an error at the output of a neural network into layer-wise and plausible 'targets' for every unit. These targets can then be used to produce weight updates for network training. However, thus far, target propagation has been heuristically proposed without demonstrable equivalence to backpropagation. Here, we derive an exact correspondence between backpropagation and a modified form of target propagation (GAIT-prop) where the target is a small perturbation of the forward pass. Specifically, backpropagation and GAIT-prop give identical updates when synaptic weight matrices are orthogonal. In a series of simple computer vision experiments, we show near-identical performance between backpropagation and GAIT-prop with a soft orthogonality-inducing regularizer.

Related papers

• Learning a Generative Meta-Model of LLM Activations [75.30161960337892]
  We create "meta-models" that learn the distribution of a network's internal states.<n>Applying the meta-model's learned prior to steering interventions improves fluency, with larger gains as loss decreases.<n>These results suggest generative meta-models offer a scalable path toward interpretability without restrictive structural assumptions.
  arXiv  Detail & Related papers  (2026-02-06T18:59:56Z)
• Weight transport through spike timing for robust local gradients [0.5236468296934584]
  plasticity in functional neural networks is frequently expressed as gradient descent on a cost. This imposes symmetry constraints that are difficult to reconcile with local computation. We introduce spike-based alignment learning, which uses spike timing statistics to extract and correct the asymmetry between effective reciprocal connections.
  arXiv  Detail & Related papers  (2025-03-04T14:05:39Z)
• Brain-like variational inference [0.0]
  Inference in both brains and machines can be formalized by maximizing the evidence lower bound (ELBO) in machine learning, or minimizing variational free energy (F) in neuroscience (ELBO = -F)<n>Here, we show that online natural gradient descent on F, under Poisson assumptions, leads to a recurrent spiking neural network that performs variational inference via membrane potential dynamics.<n>The resulting model -- the iterative Poisson variational autoencoder (iP-VAE) -- replaces the encoder network with local updates derived from natural gradient descent on F.
  arXiv  Detail & Related papers  (2024-10-25T06:00:18Z)
• PseudoNeg-MAE: Self-Supervised Point Cloud Learning using Conditional Pseudo-Negative Embeddings [55.55445978692678]
  PseudoNeg-MAE is a self-supervised learning framework that enhances global feature representation of point cloud mask autoencoders. We show that PseudoNeg-MAE achieves state-of-the-art performance on the ModelNet40 and ScanObjectNN datasets.
  arXiv  Detail & Related papers  (2024-09-24T07:57:21Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  We present Layer-wise Feedback Propagation (LFP), a novel training principle for neural network-like predictors. LFP decomposes a reward to individual neurons based on their respective contributions to solving a given task. Our method then implements a greedy approach reinforcing helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• Desire Backpropagation: A Lightweight Training Algorithm for Multi-Layer Spiking Neural Networks based on Spike-Timing-Dependent Plasticity [13.384228628766236]
  Spiking neural networks (SNNs) are a viable alternative to conventional artificial neural networks. We present desire backpropagation, a method to derive the desired spike activity of all neurons, including the hidden ones. We trained three-layer networks to classify MNIST and Fashion-MNIST images and reached an accuracy of 98.41% and 87.56%, respectively.
  arXiv  Detail & Related papers  (2022-11-10T08:32:13Z)
• Single-phase deep learning in cortico-cortical networks [1.7249361224827535]
  We introduce a new model, bursting cortico-cortical networks (BurstCCN), which integrates bursting activity, short-term plasticity and dendrite-targeting interneurons. Our results suggest that cortical features across sub-cellular, cellular, microcircuit and systems levels jointly single-phase efficient deep learning in the brain.
  arXiv  Detail & Related papers  (2022-06-23T15:10:57Z)
• Efficient and Robust Classification for Sparse Attacks [34.48667992227529]
  We consider perturbations bounded by the $ell$--norm, which have been shown as effective attacks in the domains of image-recognition, natural language processing, and malware-detection. We propose a novel defense method that consists of "truncation" and "adrial training" Motivated by the insights we obtain, we extend these components to neural network classifiers.
  arXiv  Detail & Related papers  (2022-01-23T21:18:17Z)
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks [79.74580058178594]
  We analyze the performance of training a pruned neural network by analyzing the geometric structure of the objective function. We show that the convex region near a desirable model with guaranteed generalization enlarges as the neural network model is pruned.
  arXiv  Detail & Related papers  (2021-10-12T01:11:07Z)
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks [14.685237010856953]
  We develop biologically plausible training mechanisms for self-supervised learning (SSL) in deep networks. We show that learning can be performed with one of two more plausible alternatives to backpagation.
  arXiv  Detail & Related papers  (2021-09-30T12:56:57Z)
• Relaxing the Constraints on Predictive Coding Models [62.997667081978825]
  Predictive coding is an influential theory of cortical function which posits that the principal computation the brain performs is the minimization of prediction errors. Standard implementations of the algorithm still involve potentially neurally implausible features such as identical forward and backward weights, backward nonlinear derivatives, and 1-1 error unit connectivity. In this paper, we show that these features are not integral to the algorithm and can be removed either directly or through learning additional sets of parameters with Hebbian update rules without noticeable harm to learning performance.
  arXiv  Detail & Related papers  (2020-10-02T15:21:37Z)
• Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain [62.997667081978825]
  Activation Relaxation (AR) is motivated by constructing the backpropagation gradient as the equilibrium point of a dynamical system. Our algorithm converges rapidly and robustly to the correct backpropagation gradients, requires only a single type of computational unit, and can operate on arbitrary computation graphs.
  arXiv  Detail & Related papers  (2020-09-11T11:56:34Z)
• Predictive Coding Approximates Backprop along Arbitrary Computation Graphs [68.8204255655161]
  We develop a strategy to translate core machine learning architectures into their predictive coding equivalents. Our models perform equivalently to backprop on challenging machine learning benchmarks. Our method raises the potential that standard machine learning algorithms could in principle be directly implemented in neural circuitry.
  arXiv  Detail & Related papers  (2020-06-07T15:35:47Z)
• Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias [65.13042449121411]
  In practice, training a network with the gradient estimates provided by EP does not scale to visual tasks harder than MNIST. We show that a bias in the gradient estimate of EP, inherent in the use of finite nudging, is responsible for this phenomenon. We apply these techniques to train an architecture with asymmetric forward and backward connections, yielding a 13.2% test error.
  arXiv  Detail & Related papers  (2020-06-06T09:36:07Z)
• Improving the Backpropagation Algorithm with Consequentialism Weight Updates over Mini-Batches [0.40611352512781856]
  We show that it is possible to consider a multi-layer neural network as a stack of adaptive filters. We introduce a better algorithm by predicting then emending the adverse consequences of the actions that take place in BP even before they happen. Our experiments show the usefulness of our algorithm in the training of deep neural networks.
  arXiv  Detail & Related papers  (2020-03-11T08:45:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.