Perforated Backpropagation: A Neuroscience Inspired Extension to Artificial Neural Networks

• URL: http://arxiv.org/abs/2501.18018v1
• Date: Wed, 29 Jan 2025 22:09:45 GMT
• Title: Perforated Backpropagation: A Neuroscience Inspired Extension to Artificial Neural Networks
• Authors: Rorry Brenner, Laurent Itti,
• Abstract summary: The neurons of artificial neural networks were originally invented when much less was known about biological neurons than is known today. Our work explores a modification to the core neuron unit to make it more parallel to a biological neuron. The paper explores a novel system of "Perforated" backpropagation empowering the artificial neurons of deep neural networks to achieve better performance coding.
• Score: 10.346584735416089
• License:
• Abstract: The neurons of artificial neural networks were originally invented when much less was known about biological neurons than is known today. Our work explores a modification to the core neuron unit to make it more parallel to a biological neuron. The modification is made with the knowledge that biological dendrites are not simply passive activation funnels, but also compute complex non-linear functions as they transmit activation to the cell body. The paper explores a novel system of "Perforated" backpropagation empowering the artificial neurons of deep neural networks to achieve better performance coding for the same features they coded for in the original architecture. After an initial network training phase, additional "Dendrite Nodes" are added to the network and separately trained with a different objective: to correlate their output with the remaining error of the original neurons. The trained Dendrite Nodes are then frozen, and the original neurons are further trained, now taking into account the additional error signals provided by the Dendrite Nodes. The cycle of training the original neurons and then adding and training Dendrite Nodes can be repeated several times until satisfactory performance is achieved. Our algorithm was successfully added to modern state-of-the-art PyTorch networks across multiple domains, improving upon original accuracies and allowing for significant model compression without a loss in accuracy.

Related papers

• Verified Neural Compressed Sensing [58.98637799432153]
  We develop the first (to the best of our knowledge) provably correct neural networks for a precise computational task. We show that for modest problem dimensions (up to 50), we can train neural networks that provably recover a sparse vector from linear and binarized linear measurements. We show that the complexity of the network can be adapted to the problem difficulty and solve problems where traditional compressed sensing methods are not known to provably work.
  arXiv  Detail & Related papers  (2024-05-07T12:20:12Z)
• Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks [74.3099028063756]
  We develop a new method with neuronal operations based on lateral connections and Hebbian learning. We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities. Our method consistently solves for spiking neural networks with nearly zero forgetting.
  arXiv  Detail & Related papers  (2024-02-19T09:29:37Z)
• Fast gradient-free activation maximization for neurons in spiking neural networks [5.805438104063613]
  We present a framework with an efficient design for such a loop. We track changes in the optimal stimuli for artificial neurons during training. This formation of refined optimal stimuli is associated with an increase in classification accuracy.
  arXiv  Detail & Related papers  (2023-12-28T18:30:13Z)
• Synaptic Stripping: How Pruning Can Bring Dead Neurons Back To Life [0.0]
  We introduce Synaptic Stripping as a means to combat the dead neuron problem. By automatically removing problematic connections during training, we can regenerate dead neurons. We conduct several ablation studies to investigate these dynamics as a function of network width and depth.
  arXiv  Detail & Related papers  (2023-02-11T23:55:50Z)
• A Hybrid Training Algorithm for Continuum Deep Learning Neuro-Skin Neural Network [0.0]
  Deep Learning NeuroSkin Neural Network is a new type of neural network presented recently by the authors. A neuroskin is modelled using finite elements. Each element of the finite element represents a cell. It is shown that while the neuroskin cannot present the desirable response, it improves gradually to the desired level.
  arXiv  Detail & Related papers  (2023-02-03T15:54:06Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• POPPINS : A Population-Based Digital Spiking Neuromorphic Processor with Integer Quadratic Integrate-and-Fire Neurons [50.591267188664666]
  We propose a population-based digital spiking neuromorphic processor in 180nm process technology with two hierarchy populations. The proposed approach enables the developments of biomimetic neuromorphic system and various low-power, and low-latency inference processing applications.
  arXiv  Detail & Related papers  (2022-01-19T09:26:34Z)
• Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting [135.0863818867184]
  artificial neural variability (ANV) helps artificial neural networks learn some advantages from natural'' neural networks. ANV plays as an implicit regularizer of the mutual information between the training data and the learned model. It can effectively relieve overfitting, label noise memorization, and catastrophic forgetting at negligible costs.
  arXiv  Detail & Related papers  (2020-11-12T06:06:33Z)
• Training of Deep Learning Neuro-Skin Neural Network [0.0]
  Deep Learning Neuro-Skin Neural Network is a new type of neural network presented recently by the authors. A neuroskin is modelled using finite elements. Each element of the finite element represents a cell. It is shown that while the neuroskin can not present the desirable response, it improves gradually to the desired level.
  arXiv  Detail & Related papers  (2020-07-03T18:51:45Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.