HM-DF SNN: Transcending Conventional Online Learning with Advanced Training and Deployment

• URL: http://arxiv.org/abs/2410.07547v2
• Date: Wed, 07 May 2025 10:08:15 GMT
• Title: HM-DF SNN: Transcending Conventional Online Learning with Advanced Training and Deployment
• Authors: Zecheng Hao, Yifan Huang, Zijie Xu, Wenxuan Liu, Yuanhong Tang, Zhaofei Yu, Tiejun Huang,
• Abstract summary: Spiking Neural Networks (SNNs) are considered to have enormous potential in the future development of Artificial Intelligence.<n>Current online learning framework cannot tackle the inseparability problem of temporal dependent gradients.<n>We propose Hybrid Mechanism-Driven Firing (HM-DF) model, which is a family of advanced models that respectively adopt different spiking calculation schemes.
• Score: 39.6783548791379
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking Neural Networks (SNNs) are considered to have enormous potential in the future development of Artificial Intelligence due to their brain-inspired and energy-efficient properties. Compared to vanilla Spatial-Temporal Back-propagation (STBP) training methods, online training can effectively overcome the risk of GPU memory explosion. However, current online learning framework cannot tackle the inseparability problem of temporal dependent gradients and merely aim to optimize the training memory, resulting in no performance advantages compared to the STBP training models in the inference phase. To address the aforementioned challenges, we propose Hybrid Mechanism-Driven Firing (HM-DF) model, which is a family of advanced models that respectively adopt different spiking calculation schemes in the upper-region and lower-region of the firing threshold. We point out that HM-DF model can effectively separate temporal gradients and tackle the mismatch problem of surrogate gradients, as well as achieving full-stage optimization towards computation speed and memory footprint. Experimental results have demonstrated that HM-DF model can be flexibly combined with various techniques to achieve state-of-the-art performance in the field of online learning, without triggering further power consumption.

Related papers

• Implicit Neural Differential Model for Spatiotemporal Dynamics [5.1854032131971195]
  We introduce Im-PiNDiff, a novel implicit physics-integrated neural differentiable solver for stabletemporal dynamics.<n>Inspired by deep equilibrium models, Im-PiNDiff advances the state using implicit fixed-point layers, enabling robust long-term simulation.<n>Im-PiNDiff achieves superior predictive performance, enhanced numerical stability, and substantial reductions in memory and cost.
  arXiv  Detail & Related papers  (2025-04-03T04:07:18Z)
• DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs [70.91804882618243]
  This paper proposes DSMoE, a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks.<n>We implement adaptive expert routing using sigmoid activation and straight-through estimators, enabling tokens to flexibly access different aspects of model knowledge.<n>Experiments on LLaMA models demonstrate that under equivalent computational constraints, DSMoE achieves superior performance compared to existing pruning and MoE approaches.
  arXiv  Detail & Related papers  (2025-02-18T02:37:26Z)
• Harnessing Nonidealities in Analog In-Memory Computing Circuits: A Physical Modeling Approach for Neuromorphic Systems [5.582327246405357]
  In-memory computing (IMC) offers a promising solution by addressing the von Neumann bottleneck inherent in traditional deep learning accelerators.<n>This paper presents a novel approach to directly train physical models of IMC, formulated as ordinary-differential-equation (ODE)-based physical neural networks (PNNs)<n>To enable the training of large-scale networks, we propose a technique called differentiable spike-time discretization (DSTD), which reduces the computational cost of ODE-based PNNs by up to 20 times in speed and 100 times in memory.
  arXiv  Detail & Related papers  (2024-12-12T07:22:23Z)
• Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks [69.2642802272367]
  Brain-inspired neuromorphic computing with spiking neural networks (SNNs) is a promising energy-efficient computational approach. Most recent methods leverage spatial and temporal backpropagation (BP), not adhering to neuromorphic properties. We propose a novel method, online pseudo-zeroth-order (OPZO) training.
  arXiv  Detail & Related papers  (2024-07-17T12:09:00Z)
• Efficient Online Learning for Networks of Two-Compartment Spiking Neurons [23.720523101102593]
  We present a novel online learning method specifically tailored for networks of TC-LIF neurons. We also propose a refined TC-LIF neuron model called Adaptive TC-LIF, which is carefully designed to enhance temporal information integration. Our approach successfully preserves the superior sequential modeling capabilities of the TC-LIF neuron while incorporating the training efficiency and hardware friendliness of online learning.
  arXiv  Detail & Related papers  (2024-02-25T03:15:12Z)
• Parallel Spiking Unit for Efficient Training of Spiking Neural Networks [8.912926151352888]
  Spiking Neural Networks (SNNs) are used to advance artificial intelligence. SNNs are hampered by their inherent sequential computational dependency. This paper introduces the innovative Parallel Spiking Unit (PSU) and its two derivatives, the Input-aware PSU (IPSU) and Reset-aware PSU (RPSU) These variants skillfully decouple the leaky integration and firing mechanisms in spiking neurons while probabilistically managing the reset process.
  arXiv  Detail & Related papers  (2024-02-01T09:36:26Z)
• Online Calibration of Deep Learning Sub-Models for Hybrid Numerical Modeling Systems [34.50407690251862]
  We present an efficient and practical online learning approach for hybrid systems. We demonstrate that the method, called EGA for Euler Gradient Approximation, converges to the exact gradients in the limit of infinitely small time steps. Results show significant improvements over offline learning, highlighting the potential of end-to-end online learning for hybrid modeling.
  arXiv  Detail & Related papers  (2023-11-17T17:36:26Z)
• Estimating Post-Synaptic Effects for Online Training of Feed-Forward SNNs [0.27016900604393124]
  Facilitating online learning in spiking neural networks (SNNs) is a key step in developing event-based models. We propose Online Training with Postsynaptic Estimates (OTPE) for training feed-forward SNNs. We show improved scaling for multi-layer networks using a novel approximation of temporal effects on the subsequent layer's activity.
  arXiv  Detail & Related papers  (2023-11-07T16:53:39Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  We present Layer-wise Feedback Propagation (LFP), a novel training principle for neural network-like predictors. LFP decomposes a reward to individual neurons based on their respective contributions to solving a given task. Our method then implements a greedy approach reinforcing helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks [56.35403810762512]
  Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware. We study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method.
  arXiv  Detail & Related papers  (2023-02-01T04:22:59Z)
• Dual adaptive training of photonic neural networks [30.86507809437016]
  Photonic neural network (PNN) computes with photons instead of electrons to feature low latency, high energy efficiency, and high parallelism. Existing training approaches cannot address the extensive accumulation of systematic errors in large-scale PNNs. We propose dual adaptive training ( DAT) that allows the PNN model to adapt to substantial systematic errors.
  arXiv  Detail & Related papers  (2022-12-09T05:03:45Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Online Training Through Time for Spiking Neural Networks [66.7744060103562]
  Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. We propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning.
  arXiv  Detail & Related papers  (2022-10-09T07:47:56Z)
• Online Training of Spiking Recurrent Neural Networks with Phase-Change Memory Synapses [1.9809266426888898]
  Training spiking neural networks (RNNs) on dedicated neuromorphic hardware is still an open challenge. We present a simulation framework of differential-architecture arrays based on an accurate and comprehensive Phase-Change Memory (PCM) device model. We train a spiking RNN whose weights are emulated in the presented simulation framework, using a recently proposed e-prop learning rule.
  arXiv  Detail & Related papers  (2021-08-04T01:24:17Z)
• Influence Estimation and Maximization via Neural Mean-Field Dynamics [60.91291234832546]
  We propose a novel learning framework using neural mean-field (NMF) dynamics for inference and estimation problems. Our framework can simultaneously learn the structure of the diffusion network and the evolution of node infection probabilities.
  arXiv  Detail & Related papers  (2021-06-03T00:02:05Z)
• Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training [126.49572353148262]
  We study two factors in neural network training: data parallelism and sparsity. Despite their promising benefits, understanding of their effects on neural network training remains elusive.
  arXiv  Detail & Related papers  (2020-03-25T10:49:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.