Efficient Speech Command Recognition Leveraging Spiking Neural Network and Curriculum Learning-based Knowledge Distillation

• URL: http://arxiv.org/abs/2412.12858v1
• Date: Tue, 17 Dec 2024 12:38:45 GMT
• Title: Efficient Speech Command Recognition Leveraging Spiking Neural Network and Curriculum Learning-based Knowledge Distillation
• Authors: Jiaqi Wang, Liutao Yu, Liwei Huang, Chenlin Zhou, Han Zhang, Zhenxi Song, Min Zhang, Zhengyu Ma, Zhiguo Zhang,
• Abstract summary: spiking neural networks (SNNs) excel in processing temporal information by naturally utilizing embedded time sequences as time steps.<n>Recent studies have demonstrated SNNs' effectiveness in speech command recognition, achieving high performance by employing large time steps for long time sequences.<n>We propose a high-performance fully spike-driven framework termed SpikeSCR, characterized by a global-local hybrid structure for efficient representation learning.
• Score: 30.032453125056783
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The intrinsic dynamics and event-driven nature of spiking neural networks (SNNs) make them excel in processing temporal information by naturally utilizing embedded time sequences as time steps. Recent studies adopting this approach have demonstrated SNNs' effectiveness in speech command recognition, achieving high performance by employing large time steps for long time sequences. However, the large time steps lead to increased deployment burdens for edge computing applications. Thus, it is important to balance high performance and low energy consumption when detecting temporal patterns in edge devices. Our solution comprises two key components. 1). We propose a high-performance fully spike-driven framework termed SpikeSCR, characterized by a global-local hybrid structure for efficient representation learning, which exhibits long-term learning capabilities with extended time steps. 2). To further fully embrace low energy consumption, we propose an effective knowledge distillation method based on curriculum learning (KDCL), where valuable representations learned from the easy curriculum are progressively transferred to the hard curriculum with minor loss, striking a trade-off between power efficiency and high performance. We evaluate our method on three benchmark datasets: the Spiking Heidelberg Dataset (SHD), the Spiking Speech Commands (SSC), and the Google Speech Commands (GSC) V2. Our experimental results demonstrate that SpikeSCR outperforms current state-of-the-art (SOTA) methods across these three datasets with the same time steps. Furthermore, by executing KDCL, we reduce the number of time steps by 60% and decrease energy consumption by 54.8% while maintaining comparable performance to recent SOTA results. Therefore, this work offers valuable insights for tackling temporal processing challenges with long time sequences in edge neuromorphic computing systems.

Related papers

• Echo Flow Networks [4.298381633106637]
  We introduce Echo Flow Networks (EFNs), a framework composed of a group of Echo State Networks (X-ESNs) with nonlinear readouts.<n>EFNs achieve up to 4x faster training and 3x smaller model size compared to leading methods like PatchTST.<n>One instantiation of our framework, EchoFormer, consistently achieves new state-of-the-art performance across five benchmark datasets.
  arXiv  Detail & Related papers  (2025-09-28T23:38:55Z)
• SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learning [51.10866035483686]
  High update-to-data (UTD) ratio algorithms in reinforcement learning (RL) improve sample efficiency but incur high computational costs, limiting real-world scalability. We propose Offline Stabilization Phases for Efficient Q-Learning (SPEQ), an RL algorithm that combines low-UTD online training with periodic offline stabilization phases. During these phases, Q-functions are fine-tuned with high UTD ratios on a fixed replay buffer, reducing redundant updates on suboptimal data.
  arXiv  Detail & Related papers  (2025-01-15T09:04:19Z)
• TSkips: Efficiency Through Explicit Temporal Delay Connections in Spiking Neural Networks [8.13696328386179]
  We propose TSkips, augmenting Spiking Neural Networks with forward and backward skip connections that incorporate explicit temporal delays. These connections capture long-term-temporal architectures and facilitate better spike flow over long sequences. We demonstrate the effectiveness of our approach on four event-based datasets.
  arXiv  Detail & Related papers  (2024-11-22T18:58:18Z)
• Efficient Spatio-Temporal Signal Recognition on Edge Devices Using PointLCA-Net [0.45609532372046985]
  This paper presents an approach that combines PointNet's feature extraction with the in-memory computing capabilities and energy efficiency of neuromorphic systems fortemporal signal recognition. PointNet achieves high accuracy and significantly lower energy burden during both inference and training than comparable approaches.
  arXiv  Detail & Related papers  (2024-11-21T20:48:40Z)
• Towards Low-latency Event-based Visual Recognition with Hybrid Step-wise Distillation Spiking Neural Networks [50.32980443749865]
  Spiking neural networks (SNNs) have garnered significant attention for their low power consumption and high biologicalability. Current SNNs struggle to balance accuracy and latency in neuromorphic datasets. We propose Step-wise Distillation (HSD) method, tailored for neuromorphic datasets.
  arXiv  Detail & Related papers  (2024-09-19T06:52:34Z)
• Accelerating Neural Network Training: A Brief Review [0.5825410941577593]
  This study examines innovative approaches to expedite the training process of deep neural networks (DNN) The research utilizes sophisticated methodologies, including Gradient Accumulation (GA), Automatic Mixed Precision (AMP), and Pin Memory (PM)
  arXiv  Detail & Related papers  (2023-12-15T18:43:45Z)
• Continual Learning with Dynamic Sparse Training: Exploring Algorithms for Effective Model Updates [13.983410740333788]
  Continual learning (CL) refers to the ability of an intelligent system to sequentially acquire and retain knowledge from a stream of data with as little computational overhead as possible. Dynamic Sparse Training (DST) is a prominent way to find these sparse networks and isolate them for each task. This paper is the first empirical study investigating the effect of different DST components under the CL paradigm.
  arXiv  Detail & Related papers  (2023-08-28T18:31:09Z)
• S-TLLR: STDP-inspired Temporal Local Learning Rule for Spiking Neural Networks [7.573297026523597]
  Spiking Neural Networks (SNNs) are biologically plausible models that have been identified as potentially apt for deploying energy-efficient intelligence at the edge. We propose S-TLLR, a novel three-factor temporal local learning rule inspired by the Spike-Timing Dependent Plasticity (STDP) mechanism. S-TLLR is designed to have low memory and time complexities, which are independent of the number of time steps, rendering it suitable for online learning on low-power edge devices.
  arXiv  Detail & Related papers  (2023-06-27T05:44:56Z)
• Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks [70.75043144299168]
  Spiking Neural Networks (SNNs) are promising energy-efficient models for neuromorphic computing. We propose the Spatial Learning Through Time (SLTT) method that can achieve high performance while greatly improving training efficiency. Our method achieves state-of-the-art accuracy on ImageNet, while the memory cost and training time are reduced by more than 70% and 50%, respectively, compared with BPTT.
  arXiv  Detail & Related papers  (2023-02-28T05:01:01Z)
• Braille Letter Reading: A Benchmark for Spatio-Temporal Pattern Recognition on Neuromorphic Hardware [50.380319968947035]
  Recent deep learning approaches have reached accuracy in such tasks, but their implementation on conventional embedded solutions is still computationally very and energy expensive. We propose a new benchmark for computing tactile pattern recognition at the edge through letters reading. We trained and compared feed-forward and recurrent spiking neural networks (SNNs) offline using back-propagation through time with surrogate gradients, then we deployed them on the Intel Loihimorphic chip for efficient inference. Our results show that the LSTM outperforms the recurrent SNN in terms of accuracy by 14%. However, the recurrent SNN on Loihi is 237 times more energy
  arXiv  Detail & Related papers  (2022-05-30T14:30:45Z)
• Improved Speech Emotion Recognition using Transfer Learning and Spectrogram Augmentation [56.264157127549446]
  Speech emotion recognition (SER) is a challenging task that plays a crucial role in natural human-computer interaction. One of the main challenges in SER is data scarcity. We propose a transfer learning strategy combined with spectrogram augmentation.
  arXiv  Detail & Related papers  (2021-08-05T10:39:39Z)
• Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition [79.60708268515293]
  This paper explores how to train small and efficient networks for action recognition. We propose two distillation strategies in the frequency domain, namely the feature spectrum and parameter distribution distillations respectively. Our method can achieve higher performance than state-of-the-art methods with the same backbone.
  arXiv  Detail & Related papers  (2020-09-15T07:29:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.