A Self-Ensemble Inspired Approach for Effective Training of Binary-Weight Spiking Neural Networks

• URL: http://arxiv.org/abs/2508.12609v1
• Date: Mon, 18 Aug 2025 04:11:06 GMT
• Title: A Self-Ensemble Inspired Approach for Effective Training of Binary-Weight Spiking Neural Networks
• Authors: Qingyan Meng, Mingqing Xiao, Zhengyu Ma, Huihui Zhou, Yonghong Tian, Zhouchen Lin,
• Abstract summary: Training Spiking Neural Networks (SNNs) and Binary Neural Networks (BNNs) is challenging because of the non-differentiable spike generation function.<n>We present a novel perspective on the dynamics of SNNs and their close connection to BNNs through an analysis of the backpropagation process.<n>Specifically, we leverage a structure of multiple shortcuts and a knowledge distillation-based training technique to improve the training of (binary-weight) SNNs.
• Score: 66.80058515743468
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking Neural Networks (SNNs) are a promising approach to low-power applications on neuromorphic hardware due to their energy efficiency. However, training SNNs is challenging because of the non-differentiable spike generation function. To address this issue, the commonly used approach is to adopt the backpropagation through time framework, while assigning the gradient of the non-differentiable function with some surrogates. Similarly, Binary Neural Networks (BNNs) also face the non-differentiability problem and rely on approximating gradients. However, the deep relationship between these two fields and how their training techniques can benefit each other has not been systematically researched. Furthermore, training binary-weight SNNs is even more difficult. In this work, we present a novel perspective on the dynamics of SNNs and their close connection to BNNs through an analysis of the backpropagation process. We demonstrate that training a feedforward SNN can be viewed as training a self-ensemble of a binary-activation neural network with noise injection. Drawing from this new understanding of SNN dynamics, we introduce the Self-Ensemble Inspired training method for (Binary-Weight) SNNs (SEI-BWSNN), which achieves high-performance results with low latency even for the case of the 1-bit weights. Specifically, we leverage a structure of multiple shortcuts and a knowledge distillation-based training technique to improve the training of (binary-weight) SNNs. Notably, by binarizing FFN layers in a Transformer architecture, our approach achieves 82.52% accuracy on ImageNet with only 2 time steps, indicating the effectiveness of our methodology and the potential of binary-weight SNNs.

Related papers

• S$^2$NN: Sub-bit Spiking Neural Networks [53.08060832135342]
  Spiking Neural Networks (SNNs) offer an energy-efficient paradigm for machine intelligence.<n>Despite recent advances in binary SNNs, the storage and computational demands remain substantial for large-scale networks.<n>We propose Sub-bit Spiking Neural Networks (S$2$NNs) that represent weights with less than one bit.
  arXiv  Detail & Related papers  (2025-09-29T04:17:44Z)
• ReverB-SNN: Reversing Bit of the Weight and Activation for Spiking Neural Networks [22.66861050525175]
  Spiking Neural Network (SNN), a biologically inspired neural network infrastructure, has garnered significant attention recently.<n>We advocate reversing the bit of the weight and activation for SNNs, called textbfReverB-SNN.<n>This preserves the event-driven and multiplication-free advantages of standard SNNs while enhancing the information capacity of activations.
  arXiv  Detail & Related papers  (2025-06-09T13:02:03Z)
• Training Spiking Neural Networks via Augmented Direct Feedback Alignment [3.798885293742468]
  Spiking neural networks (SNNs) are promising solutions for implementing neural networks in neuromorphic devices. However, the nondifferentiable nature of SNN neurons makes it a challenge to train them. In this paper, we propose using augmented direct feedback alignment (aDFA), a gradient-free approach based on random projection, to train SNNs.
  arXiv  Detail & Related papers  (2024-09-12T06:22:44Z)
• BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation [20.34272550256856]
  Spiking neural networks (SNNs) mimic biological neural system to convey information via discrete spikes. Our work achieves state-of-the-art performance for training SNNs on both static and neuromorphic datasets.
  arXiv  Detail & Related papers  (2024-07-12T08:17:24Z)
• Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization [12.1610509770913]
  Spiking Neural Networks (SNNs) mimic the spiking nature of brain neurons. We propose a joint training framework of ANN and SNN, in which the ANN can guide the SNN's optimization. Our method consistently outperforms many other state-of-the-art training methods.
  arXiv  Detail & Related papers  (2023-05-03T13:12:17Z)
• Recurrent Bilinear Optimization for Binary Neural Networks [58.972212365275595]
  BNNs neglect the intrinsic bilinear relationship of real-valued weights and scale factors. Our work is the first attempt to optimize BNNs from the bilinear perspective. We obtain robust RBONNs, which show impressive performance over state-of-the-art BNNs on various models and datasets.
  arXiv  Detail & Related papers  (2022-09-04T06:45:33Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Rethinking Pretraining as a Bridge from ANNs to SNNs [13.984523794353477]
  Spiking neural networks (SNNs) are known as a typical kind of brain-inspired models with their unique features. How to obtain a high-accuracy model has always been the main challenge in the field of SNN.
  arXiv  Detail & Related papers  (2022-03-02T14:59:57Z)
• Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layer [77.78479877473899]
  We design a spatial-temporal-fusion BNN for efficiently scaling BNNs to large models. Compared to vanilla BNNs, our approach can greatly reduce the training time and the number of parameters, which contributes to scale BNNs efficiently.
  arXiv  Detail & Related papers  (2021-12-12T17:13:14Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.