Interpreting learning in biological neural networks as zero-order
optimization method
- URL: http://arxiv.org/abs/2301.11777v2
- Date: Thu, 23 Mar 2023 13:28:58 GMT
- Title: Interpreting learning in biological neural networks as zero-order
optimization method
- Authors: Johannes Schmidt-Hieber
- Abstract summary: In this work, we look at the brain as a statistical method for supervised learning.
The main contribution is to relate the local updating rule of the connection parameters in BNNs to a zero-order optimization method.
- Score: 0.0
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Recently, significant progress has been made regarding the statistical
understanding of artificial neural networks (ANNs). ANNs are motivated by the
functioning of the brain, but differ in several crucial aspects. In particular,
the locality in the updating rule of the connection parameters in biological
neural networks (BNNs) makes it biologically implausible that the learning of
the brain is based on gradient descent. In this work, we look at the brain as a
statistical method for supervised learning. The main contribution is to relate
the local updating rule of the connection parameters in BNNs to a zero-order
optimization method. It is shown that the expected values of the iterates
implement a modification of gradient descent.
Related papers
- BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation [20.34272550256856]
Spiking neural networks (SNNs) mimic biological neural system to convey information via discrete spikes.
Our work achieves state-of-the-art performance for training SNNs on both static and neuromorphic datasets.
arXiv Detail & Related papers (2024-07-12T08:17:24Z) - Temporal Spiking Neural Networks with Synaptic Delay for Graph Reasoning [91.29876772547348]
Spiking neural networks (SNNs) are investigated as biologically inspired models of neural computation.
This paper reveals that SNNs, when amalgamated with synaptic delay and temporal coding, are proficient in executing (knowledge) graph reasoning.
arXiv Detail & Related papers (2024-05-27T05:53:30Z) - Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
We propose to represent neural networks as computational graphs of parameters.
Our approach enables a single model to encode neural computational graphs with diverse architectures.
We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
arXiv Detail & Related papers (2024-03-18T18:01:01Z) - Hebbian Learning based Orthogonal Projection for Continual Learning of
Spiking Neural Networks [74.3099028063756]
We develop a new method with neuronal operations based on lateral connections and Hebbian learning.
We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities.
Our method consistently solves for spiking neural networks with nearly zero forgetting.
arXiv Detail & Related papers (2024-02-19T09:29:37Z) - Is Learning in Biological Neural Networks based on Stochastic Gradient Descent? An analysis using stochastic processes [0.0]
We study a model for supervised learning in biological neural networks (BNNs)
We show that a gradient step occurs approximately when each learning opportunity is processed by many local updates.
This result suggests that gradient descent may indeed play a role in optimizing BNNs.
arXiv Detail & Related papers (2023-09-10T18:12:52Z) - aSTDP: A More Biologically Plausible Learning [0.0]
We introduce approximate STDP, a new neural networks learning framework.
It uses only STDP rules for supervised and unsupervised learning.
It can make predictions or generate patterns in one model without additional configuration.
arXiv Detail & Related papers (2022-05-22T08:12:50Z) - Biologically-inspired neuronal adaptation improves learning in neural
networks [0.7734726150561086]
Humans still outperform artificial neural networks on many tasks.
We draw inspiration from the brain to improve machine learning algorithms.
We add adaptation to multilayer perceptrons and convolutional neural networks trained on MNIST and CIFAR-10.
arXiv Detail & Related papers (2022-04-08T16:16:02Z) - Training Feedback Spiking Neural Networks by Implicit Differentiation on
the Equilibrium State [66.2457134675891]
Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware.
Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks.
We propose a novel training method that does not rely on the exact reverse of the forward computation.
arXiv Detail & Related papers (2021-09-29T07:46:54Z) - Dynamic Neural Diversification: Path to Computationally Sustainable
Neural Networks [68.8204255655161]
Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks.
We explore the diversity of the neurons within the hidden layer during the learning process.
We analyze how the diversity of the neurons affects predictions of the model.
arXiv Detail & Related papers (2021-09-20T15:12:16Z) - Progressive Tandem Learning for Pattern Recognition with Deep Spiking
Neural Networks [80.15411508088522]
Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency.
We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
arXiv Detail & Related papers (2020-07-02T15:38:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.