Is Learning in Biological Neural Networks based on Stochastic Gradient Descent? An analysis using stochastic processes

• URL: http://arxiv.org/abs/2309.05102v3
• Date: Wed, 10 Apr 2024 15:02:35 GMT
• Title: Is Learning in Biological Neural Networks based on Stochastic Gradient Descent? An analysis using stochastic processes
• Authors: Sören Christensen, Jan Kallsen,
• Abstract summary: We study a model for supervised learning in biological neural networks (BNNs) We show that a gradient step occurs approximately when each learning opportunity is processed by many local updates. This result suggests that gradient descent may indeed play a role in optimizing BNNs.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, there has been an intense debate about how learning in biological neural networks (BNNs) differs from learning in artificial neural networks. It is often argued that the updating of connections in the brain relies only on local information, and therefore a stochastic gradient-descent type optimization method cannot be used. In this paper, we study a stochastic model for supervised learning in BNNs. We show that a (continuous) gradient step occurs approximately when each learning opportunity is processed by many local updates. This result suggests that stochastic gradient descent may indeed play a role in optimizing BNNs.

Related papers

• ADMM-Based Training for Spiking Neural Networks [1.1249583407496218]
  spiking neural networks (SNNs) have gained momentum due to their high potential in time-series processing combined with minimal energy consumption.<n>They still lack a dedicated and efficient training algorithm.<n>We propose a novel SNN training method based on the alternating direction method of multipliers (ADMM)
  arXiv  Detail & Related papers  (2025-05-08T10:20:33Z)
• Randomized Forward Mode Gradient for Spiking Neural Networks in Scientific Machine Learning [4.178826560825283]
  Spiking neural networks (SNNs) represent a promising approach in machine learning, combining the hierarchical learning capabilities of deep neural networks with the energy efficiency of spike-based computations. Traditional end-to-end training of SNNs is often based on back-propagation, where weight updates are derived from gradients computed through the chain rule. This method encounters challenges due to its limited biological plausibility and inefficiencies on neuromorphic hardware. In this study, we introduce an alternative training approach for SNNs. Instead of using back-propagation, we leverage weight perturbation methods within a forward-mode
  arXiv  Detail & Related papers  (2024-11-11T15:20:54Z)
• Fractional-order spike-timing-dependent gradient descent for multi-layer spiking neural networks [18.142378139047977]
  This paper proposes a fractional-order spike-timing-dependent gradient descent (FOSTDGD) learning model. It is tested on theNIST and DVS128 Gesture datasets and its accuracy under different network structure and fractional orders is analyzed.
  arXiv  Detail & Related papers  (2024-10-20T05:31:34Z)
• BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation [20.34272550256856]
  Spiking neural networks (SNNs) mimic biological neural system to convey information via discrete spikes. Our work achieves state-of-the-art performance for training SNNs on both static and neuromorphic datasets.
  arXiv  Detail & Related papers  (2024-07-12T08:17:24Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Interpreting learning in biological neural networks as zero-order optimization method [0.0]
  In this work, we look at the brain as a statistical method for supervised learning. The main contribution is to relate the local updating rule of the connection parameters in BNNs to a zero-order optimization method.
  arXiv  Detail & Related papers  (2023-01-27T15:30:25Z)
• Exact Gradient Computation for Spiking Neural Networks Through Forward Propagation [39.33537954568678]
  Spiking neural networks (SNN) have emerged as alternatives to traditional neural networks. We propose a novel training algorithm, called emphforward propagation (FP), that computes exact gradients for SNN.
  arXiv  Detail & Related papers  (2022-10-18T20:28:21Z)
• Transfer Learning with Deep Tabular Models [66.67017691983182]
  We show that upstream data gives tabular neural networks a decisive advantage over GBDT models. We propose a realistic medical diagnosis benchmark for tabular transfer learning. We propose a pseudo-feature method for cases where the upstream and downstream feature sets differ.
  arXiv  Detail & Related papers  (2022-06-30T14:24:32Z)
• Can we learn gradients by Hamiltonian Neural Networks? [68.8204255655161]
  We propose a meta-learner based on ODE neural networks that learns gradients. We demonstrate that our method outperforms a meta-learner based on LSTM for an artificial task and the MNIST dataset with ReLU activations in the optimizee.
  arXiv  Detail & Related papers  (2021-10-31T18:35:10Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.