Single-phase deep learning in cortico-cortical networks

• URL: http://arxiv.org/abs/2206.11769v1
• Date: Thu, 23 Jun 2022 15:10:57 GMT
• Title: Single-phase deep learning in cortico-cortical networks
• Authors: Will Greedy, Heng Wei Zhu, Joseph Pemberton, Jack Mellor and Rui Ponte Costa
• Abstract summary: We introduce a new model, bursting cortico-cortical networks (BurstCCN), which integrates bursting activity, short-term plasticity and dendrite-targeting interneurons. Our results suggest that cortical features across sub-cellular, cellular, microcircuit and systems levels jointly single-phase efficient deep learning in the brain.
• Score: 1.7249361224827535
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The error-backpropagation (backprop) algorithm remains the most common solution to the credit assignment problem in artificial neural networks. In neuroscience, it is unclear whether the brain could adopt a similar strategy to correctly modify its synapses. Recent models have attempted to bridge this gap while being consistent with a range of experimental observations. However, these models are either unable to effectively backpropagate error signals across multiple layers or require a multi-phase learning process, neither of which are reminiscent of learning in the brain. Here, we introduce a new model, bursting cortico-cortical networks (BurstCCN), which solves these issues by integrating known properties of cortical networks namely bursting activity, short-term plasticity (STP) and dendrite-targeting interneurons. BurstCCN relies on burst multiplexing via connection-type-specific STP to propagate backprop-like error signals within deep cortical networks. These error signals are encoded at distal dendrites and induce burst-dependent plasticity as a result of excitatory-inhibitory topdown inputs. First, we demonstrate that our model can effectively backpropagate errors through multiple layers using a single-phase learning process. Next, we show both empirically and analytically that learning in our model approximates backprop-derived gradients. Finally, we demonstrate that our model is capable of learning complex image classification tasks (MNIST and CIFAR-10). Overall, our results suggest that cortical features across sub-cellular, cellular, microcircuit and systems levels jointly underlie single-phase efficient deep learning in the brain.

Related papers

• Dis-inhibitory neuronal circuits can control the sign of synaptic plasticity [6.227678387562755]
  We show that error-modulated learning emerges naturally at the circuit level when recurrent inhibition explicitly influences Hebbian plasticity. Our findings bridge the gap between functional and experimentally observed plasticity rules.
  arXiv  Detail & Related papers  (2023-10-30T15:06:19Z)
• Unsupervised Spiking Neural Network Model of Prefrontal Cortex to study Task Switching with Synaptic deficiency [0.0]
  We build a computational model of Prefrontal Cortex (PFC) using Spiking Neural Networks (SNN) In this study, we use SNN's having parameters close to biologically plausible values and train the model using unsupervised Spike Timing Dependent Plasticity (STDP) learning rule.
  arXiv  Detail & Related papers  (2023-05-23T05:59:54Z)
• Learning efficient backprojections across cortical hierarchies in real time [1.6474865533365743]
  We introduce a bio-plausible method to learn efficient feedback weights in layered cortical hierarchies. All weights are learned simultaneously with always-on plasticity and using only information locally available to the synapses. Our method is applicable to a wide class of models and improves on previously known biologically plausible ways of credit assignment.
  arXiv  Detail & Related papers  (2022-12-20T13:54:04Z)
• Go Beyond Multiple Instance Neural Networks: Deep-learning Models based on Local Pattern Aggregation [0.0]
  convolutional neural networks (CNNs) have brought breakthroughs in processing clinical electrocardiograms (ECGs) and speaker-independent speech. In this paper, we propose local pattern aggregation-based deep-learning models to effectively deal with both problems. The novel network structure, called LPANet, has cropping and aggregation operations embedded into it.
  arXiv  Detail & Related papers  (2022-05-28T13:18:18Z)
• Neurosymbolic hybrid approach to driver collision warning [64.02492460600905]
  There are two main algorithmic approaches to autonomous driving systems. Deep learning alone has achieved state-of-the-art results in many areas. But sometimes it can be very difficult to debug if the deep learning model doesn't work.
  arXiv  Detail & Related papers  (2022-03-28T20:29:50Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Sign and Relevance Learning [0.0]
  Standard models of biologically realistic reinforcement learning employ a global error signal, which implies the use of shallow networks. In this study, we introduce a novel network that solves this problem by propagating only the sign of the plasticity change. Neuromodulation can be understood as a rectified error or relevance signal, while the top-down sign of the error signal determines whether long-term depression will occur.
  arXiv  Detail & Related papers  (2021-10-14T11:57:57Z)
• Understanding Self-supervised Learning with Dual Deep Networks [74.92916579635336]
  We propose a novel framework to understand contrastive self-supervised learning (SSL) methods that employ dual pairs of deep ReLU networks. We prove that in each SGD update of SimCLR with various loss functions, the weights at each layer are updated by a emphcovariance operator. To further study what role the covariance operator plays and which features are learned in such a process, we model data generation and augmentation processes through a emphhierarchical latent tree model (HLTM)
  arXiv  Detail & Related papers  (2020-10-01T17:51:49Z)
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures [179.66117325866585]
  We investigate a design space that is usually overlooked, i.e. adjusting the channel configurations of predefined networks. We find that this adjustment can be achieved by shrinking widened baseline networks and leads to superior performance. Experiments are conducted on various networks and datasets for image classification, visual tracking and image restoration.
  arXiv  Detail & Related papers  (2020-06-29T17:59:26Z)
• Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks [55.0627904986664]
  Spiking Neural Networks (SNNs) usetemporal spike patterns to represent and transmit information, which is not only biologically realistic but also suitable for ultra-low-power event-driven neuromorphic implementation. This paper investigates the contribution of spike timing dynamics to information encoding, synaptic plasticity and decision making, providing a new perspective to design of future DeepSNNs and neuromorphic hardware systems.
  arXiv  Detail & Related papers  (2020-03-26T11:13:07Z)
• Belief Propagation Reloaded: Learning BP-Layers for Labeling Problems [83.98774574197613]
  We take one of the simplest inference methods, a truncated max-product Belief propagation, and add what is necessary to make it a proper component of a deep learning model. This BP-Layer can be used as the final or an intermediate block in convolutional neural networks (CNNs) The model is applicable to a range of dense prediction problems, is well-trainable and provides parameter-efficient and robust solutions in stereo, optical flow and semantic segmentation.
  arXiv  Detail & Related papers  (2020-03-13T13:11:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.