Flexible and Scalable Deep Dendritic Spiking Neural Networks with Multiple Nonlinear Branching

• URL: http://arxiv.org/abs/2412.06355v1
• Date: Mon, 09 Dec 2024 10:15:46 GMT
• Title: Flexible and Scalable Deep Dendritic Spiking Neural Networks with Multiple Nonlinear Branching
• Authors: Yifan Huang, Wei Fang, Zhengyu Ma, Guoqi Li, Yonghong Tian,
• Abstract summary: We propose the dendritic spiking neuron (DendSN) incorporating multiple dendritic branches with nonlinear dynamics. Compared to the point spiking neurons, DendSN exhibits significantly higher expressivity. Our work demonstrates the possibility of training bio-plausible dendritic SNNs with depths and scales comparable to traditional point SNNs.
• Score: 39.664692909673086
• License:
• Abstract: Recent advances in spiking neural networks (SNNs) have a predominant focus on network architectures, while relatively little attention has been paid to the underlying neuron model. The point neuron models, a cornerstone of deep SNNs, pose a bottleneck on the network-level expressivity since they depict somatic dynamics only. In contrast, the multi-compartment models in neuroscience offer remarkable expressivity by introducing dendritic morphology and dynamics, but remain underexplored in deep learning due to their unaffordable computational cost and inflexibility. To combine the advantages of both sides for a flexible, efficient yet more powerful model, we propose the dendritic spiking neuron (DendSN) incorporating multiple dendritic branches with nonlinear dynamics. Compared to the point spiking neurons, DendSN exhibits significantly higher expressivity. DendSN's flexibility enables its seamless integration into diverse deep SNN architectures. To accelerate dendritic SNNs (DendSNNs), we parallelize dendritic state updates across time steps, and develop Triton kernels for GPU-level acceleration. As a result, we can construct large-scale DendSNNs with depth comparable to their point SNN counterparts. Next, we comprehensively evaluate DendSNNs' performance on various demanding tasks. By modulating dendritic branch strengths using a context signal, catastrophic forgetting of DendSNNs is substantially mitigated. Moreover, DendSNNs demonstrate enhanced robustness against noise and adversarial attacks compared to point SNNs, and excel in few-shot learning settings. Our work firstly demonstrates the possibility of training bio-plausible dendritic SNNs with depths and scales comparable to traditional point SNNs, and reveals superior expressivity and robustness of reduced dendritic neuron models in deep learning, thereby offering a fresh perspective on advancing neural network design.

Related papers

• Direct Training High-Performance Deep Spiking Neural Networks: A Review of Theories and Methods [33.377770671553336]
  Spiking neural networks (SNNs) offer a promising energy-efficient alternative to artificial neural networks (ANNs) In this paper, we provide a new perspective to summarize the theories and methods for training deep SNNs with high performance.
  arXiv  Detail & Related papers  (2024-05-06T09:58:54Z)
• SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence [51.6943465041708]
  Spiking neural networks (SNNs) aim to realize brain-inspired intelligence on neuromorphic chips with high energy efficiency. We contribute a full-stack toolkit for pre-processing neuromorphic datasets, building deep SNNs, optimizing their parameters, and deploying SNNs on neuromorphic chips.
  arXiv  Detail & Related papers  (2023-10-25T13:15:17Z)
• A Hybrid Neural Coding Approach for Pattern Recognition with Spiking Neural Networks [53.31941519245432]
  Brain-inspired spiking neural networks (SNNs) have demonstrated promising capabilities in solving pattern recognition tasks. These SNNs are grounded on homogeneous neurons that utilize a uniform neural coding for information representation. In this study, we argue that SNN architectures should be holistically designed to incorporate heterogeneous coding schemes.
  arXiv  Detail & Related papers  (2023-05-26T02:52:12Z)
• Fluctuation-driven initialization for spiking neural network training [3.976291254896486]
  Spiking neural networks (SNNs) underlie low-power, fault-tolerant information processing in the brain. We develop a general strategy for SNNs inspired by the fluctuation-driven regime commonly observed in the brain.
  arXiv  Detail & Related papers  (2022-06-21T09:48:49Z)
• On the Intrinsic Structures of Spiking Neural Networks [66.57589494713515]
  Recent years have emerged a surge of interest in SNNs owing to their remarkable potential to handle time-dependent and event-driven data. There has been a dearth of comprehensive studies examining the impact of intrinsic structures within spiking computations. This work delves deep into the intrinsic structures of SNNs, by elucidating their influence on the expressivity of SNNs.
  arXiv  Detail & Related papers  (2022-06-21T09:42:30Z)
• A Synapse-Threshold Synergistic Learning Approach for Spiking Neural Networks [1.8556712517882232]
  Spiking neural networks (SNNs) have demonstrated excellent capabilities in various intelligent scenarios. In this study, we develop a novel synergistic learning approach that involves simultaneously training synaptic weights and spike thresholds in SNNs.
  arXiv  Detail & Related papers  (2022-06-10T06:41:36Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• BackEISNN: A Deep Spiking Neural Network with Adaptive Self-Feedback and Balanced Excitatory-Inhibitory Neurons [8.956708722109415]
  Spiking neural networks (SNNs) transmit information through discrete spikes, which performs well in processing spatial-temporal information. We propose a deep spiking neural network with adaptive self-feedback and balanced excitatory and inhibitory neurons (BackEISNN) For the MNIST, FashionMNIST, and N-MNIST datasets, our model has achieved state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-05-27T08:38:31Z)
• Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks [1.8515971640245998]
  Spiking neural networks (SNNs) have been investigated as more biologically plausible and potentially more powerful models of neural computation. We show how a novel surrogate gradient combined with recurrent networks of tunable and adaptive spiking neurons yields state-of-the-art for SNNs.
  arXiv  Detail & Related papers  (2021-03-12T10:27:29Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.