LLS: Local Learning Rule for Deep Neural Networks Inspired by Neural Activity Synchronization

• URL: http://arxiv.org/abs/2405.15868v2
• Date: Tue, 29 Oct 2024 16:35:59 GMT
• Title: LLS: Local Learning Rule for Deep Neural Networks Inspired by Neural Activity Synchronization
• Authors: Marco Paul E. Apolinario, Arani Roy, Kaushik Roy,
• Abstract summary: Training deep neural networks (DNNs) using traditional backpropagation (BP) presents challenges in terms of computational complexity and energy consumption. We propose a novel Local Learning rule inspired by neural activity Synchronization phenomena (LLS) observed in the brain. LLS achieves comparable performance with up to $300 times$ fewer multiply-accumulate (MAC) operations and half the memory requirements of BP.
• Score: 6.738409533239947
• License:
• Abstract: Training deep neural networks (DNNs) using traditional backpropagation (BP) presents challenges in terms of computational complexity and energy consumption, particularly for on-device learning where computational resources are limited. Various alternatives to BP, including random feedback alignment, forward-forward, and local classifiers, have been explored to address these challenges. These methods have their advantages, but they can encounter difficulties when dealing with intricate visual tasks or demand considerable computational resources. In this paper, we propose a novel Local Learning rule inspired by neural activity Synchronization phenomena (LLS) observed in the brain. LLS utilizes fixed periodic basis vectors to synchronize neuron activity within each layer, enabling efficient training without the need for additional trainable parameters. We demonstrate the effectiveness of LLS and its variations, LLS-M and LLS-MxM, on multiple image classification datasets, achieving accuracy comparable to BP with reduced computational complexity and minimal additional parameters. Specifically, LLS achieves comparable performance with up to $300 \times$ fewer multiply-accumulate (MAC) operations and half the memory requirements of BP. Furthermore, the performance of LLS on the Visual Wake Word (VWW) dataset highlights its suitability for on-device learning tasks, making it a promising candidate for edge hardware implementations.

Related papers

• S-TLLR: STDP-inspired Temporal Local Learning Rule for Spiking Neural Networks [7.573297026523597]
  Spiking Neural Networks (SNNs) are biologically plausible models that have been identified as potentially apt for deploying energy-efficient intelligence at the edge. We propose S-TLLR, a novel three-factor temporal local learning rule inspired by the Spike-Timing Dependent Plasticity (STDP) mechanism. S-TLLR is designed to have low memory and time complexities, which are independent of the number of time steps, rendering it suitable for online learning on low-power edge devices.
  arXiv  Detail & Related papers  (2023-06-27T05:44:56Z)
• Properties and Potential Applications of Random Functional-Linked Types of Neural Networks [81.56822938033119]
  Random functional-linked neural networks (RFLNNs) offer an alternative way of learning in deep structure. This paper gives some insights into the properties of RFLNNs from the viewpoints of frequency domain. We propose a method to generate a BLS network with better performance, and design an efficient algorithm for solving Poison's equation.
  arXiv  Detail & Related papers  (2023-04-03T13:25:22Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Towards Energy-Efficient, Low-Latency and Accurate Spiking LSTMs [1.7969777786551424]
  Spiking Neural Networks (SNNs) have emerged as an attractive-temporal computing paradigm vision for complex tasks. We propose an optimized spiking long short-term memory networks (LSTM) training framework that involves a novel. rev-to-SNN conversion framework, followed by SNN training. We evaluate our framework on sequential learning tasks including temporal M, Google Speech Commands (GSC) datasets, and UCI Smartphone on different LSTM architectures.
  arXiv  Detail & Related papers  (2022-10-23T04:10:27Z)
• Deep Q-network using reservoir computing with multi-layered readout [0.0]
  Recurrent neural network (RNN) based reinforcement learning (RL) is used for learning context-dependent tasks. An approach with replay memory introducing reservoir computing has been proposed, which trains an agent without BPTT. This paper shows that the performance of this method improves by using a multi-layered neural network for the readout layer.
  arXiv  Detail & Related papers  (2022-03-03T00:32:55Z)
• Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training [18.521882534906972]
  We propose to structure dropout patterns, by dropping out the same set of physical neurons within a batch, resulting in column (row) level hidden state sparsity. We conduct experiments for three representative NLP tasks: language modelling on the PTB dataset, OpenNMT based machine translation using the IWSLT De-En and En-Vi datasets, and named entity recognition sequence labelling.
  arXiv  Detail & Related papers  (2021-06-22T22:44:32Z)
• Real-time Multi-Task Diffractive Deep Neural Networks via Hardware-Software Co-design [1.6066483376871004]
  This work proposes a novel hardware-software co-design method that enables robust and noise-resilient Multi-task Learning in D$2$NNs. Our experimental results demonstrate significant improvements in versatility and hardware efficiency, and also demonstrate the robustness of proposed multi-task D$2$NN architecture.
  arXiv  Detail & Related papers  (2020-12-16T12:29:54Z)
• A journey in ESN and LSTM visualisations on a language task [77.34726150561087]
  We trained ESNs and LSTMs on a Cross-Situationnal Learning (CSL) task. The results are of three kinds: performance comparison, internal dynamics analyses and visualization of latent space.
  arXiv  Detail & Related papers  (2020-12-03T08:32:01Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• Self-Organized Operational Neural Networks with Generative Neurons [87.32169414230822]
  ONNs are heterogenous networks with a generalized neuron model that can encapsulate any set of non-linear operators. We propose Self-organized ONNs (Self-ONNs) with generative neurons that have the ability to adapt (optimize) the nodal operator of each connection.
  arXiv  Detail & Related papers  (2020-04-24T14:37:56Z)
• Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks [55.0627904986664]
  Spiking Neural Networks (SNNs) usetemporal spike patterns to represent and transmit information, which is not only biologically realistic but also suitable for ultra-low-power event-driven neuromorphic implementation. This paper investigates the contribution of spike timing dynamics to information encoding, synaptic plasticity and decision making, providing a new perspective to design of future DeepSNNs and neuromorphic hardware systems.
  arXiv  Detail & Related papers  (2020-03-26T11:13:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.