Compressed Latent Replays for Lightweight Continual Learning on Spiking Neural Networks

• URL: http://arxiv.org/abs/2407.03111v2
• Date: Thu, 4 Jul 2024 08:07:18 GMT
• Title: Compressed Latent Replays for Lightweight Continual Learning on Spiking Neural Networks
• Authors: Alberto Dequino, Alessio Carpegna, Davide Nadalini, Alessandro Savino, Luca Benini, Stefano Di Carlo, Francesco Conti,
• Abstract summary: We introduce the first memory-efficient implementation of Latent Replay (LR)-based CL for Spiking Neural Networks (SNNs) LRs combine new samples with latent representations of previously learned data, to mitigate forgetting. Experiments on the Heidelberg SHD dataset with Sample and Class-Incremental tasks reach a Top-1 accuracy of 92.5% and 92%, respectively.
• Score: 45.53312220335389
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Rehearsal-based Continual Learning (CL) has been intensely investigated in Deep Neural Networks (DNNs). However, its application in Spiking Neural Networks (SNNs) has not been explored in depth. In this paper we introduce the first memory-efficient implementation of Latent Replay (LR)-based CL for SNNs, designed to seamlessly integrate with resource-constrained devices. LRs combine new samples with latent representations of previously learned data, to mitigate forgetting. Experiments on the Heidelberg SHD dataset with Sample and Class-Incremental tasks reach a Top-1 accuracy of 92.5% and 92%, respectively, without forgetting the previously learned information. Furthermore, we minimize the LRs' requirements by applying a time-domain compression, reducing by two orders of magnitude their memory requirement, with respect to a naive rehearsal setup, with a maximum accuracy drop of 4%. On a Multi-Class-Incremental task, our SNN learns 10 new classes from an initial set of 10, reaching a Top-1 accuracy of 78.4% on the full test set.

Related papers

• SAFA-SNN: Sparsity-Aware On-Device Few-Shot Class-Incremental Learning with Fast-Adaptive Structure of Spiking Neural Network [19.73335869722781]
  Continuous learning of novel classes is crucial for edge devices to preserve data privacy and maintain reliable performance in dynamic environments.<n>In this work, we present an SNN-based method for On-Device FSCIL ie., Sparsity-Aware and Fast Adaptive SNN.
  arXiv  Detail & Related papers  (2025-10-04T03:21:31Z)
• A Recovery Guarantee for Sparse Neural Networks [55.65046795628564]
  We prove the first guarantees of sparse recovery for ReLU neural networks.<n>We find performance that is competitive with, and often exceeds, a high-performing but memory-inefficient baseline.
  arXiv  Detail & Related papers  (2025-09-24T17:10:48Z)
• Online Continual Learning via Spiking Neural Networks with Sleep Enhanced Latent Replay [8.108335297331658]
  This paper proposes a novel online continual learning approach termed as SESLR.<n>It incorporates a sleep enhanced latent replay scheme with spiking neural networks (SNNs)<n>Experiments on both conventional (MNIST, CIFAR10) and neuromorphic (NMNIST, CIFAR10-DVS) datasets demonstrate SESLR's effectiveness.
  arXiv  Detail & Related papers  (2025-06-23T12:22:39Z)
• Continual Learning with Columnar Spiking Neural Networks [0.0]
  We show that microcolumns adapt most efficiently to new tasks when they lack shared structure with prior learning.<n>Our optimal configuration learns ten sequential MNIST tasks effectively, maintaining 92% accuracy on each.<n>It shows low forgetting, with only 4% performance degradation on the first task after training on nine subsequent tasks.
  arXiv  Detail & Related papers  (2025-06-20T17:13:38Z)
• STOP: Spatiotemporal Orthogonal Propagation for Weight-Threshold-Leakage Synergistic Training of Deep Spiking Neural Networks [11.85044871205734]
  Deep neural network (SNN) models based on sparsely sparse binary activations lack efficient and high-accuracy SNN deep learning algorithms. Our algorithm enables fully synergistic learning algorithm firing synaptic weights as well as thresholds and spiking factors in neurons to improve SNN accuracy. Under a unified temporally-forward trace-based framework, we mitigate the huge memory requirement for storing neural states of all time-steps in the forward pass. Our method is more plausible for edge intelligent scenarios where resources are limited but high-accuracy in-situ learning is desired.
  arXiv  Detail & Related papers  (2024-11-17T14:15:54Z)
• Boosting Binary Neural Networks via Dynamic Thresholds Learning [21.835748440099586]
  We introduce DySign to reduce information loss and boost representative capacity of BNNs. For DCNNs, DyBCNNs based on two backbones achieve 71.2% and 67.4% top1-accuracy on ImageNet dataset. For ViTs, DyCCT presents the superiority of the convolutional embedding layer in fully binarized ViTs and 56.1% on the ImageNet dataset.
  arXiv  Detail & Related papers  (2022-11-04T07:18:21Z)
• Training Spiking Neural Networks with Local Tandem Learning [96.32026780517097]
  Spiking neural networks (SNNs) are shown to be more biologically plausible and energy efficient than their predecessors. In this paper, we put forward a generalized learning rule, termed Local Tandem Learning (LTL) We demonstrate rapid network convergence within five training epochs on the CIFAR-10 dataset while having low computational complexity.
  arXiv  Detail & Related papers  (2022-10-10T10:05:00Z)
• Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning [54.7584721943286]
  Continual Learning (CL) methods aim to enable machine learning models to learn new tasks without catastrophic forgetting of those that have been previously mastered. Existing CL approaches often keep a buffer of previously-seen samples, perform knowledge distillation, or use regularization techniques towards this goal. We propose to only activate and select sparse neurons for learning current and past tasks at any stage.
  arXiv  Detail & Related papers  (2022-02-21T13:25:03Z)
• Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learning [6.417011237981518]
  Spiking neural networks (SNNs) with leaky integrate and fire (LIF) neurons can be operated in an event-driven manner. We show that SNNs can be trained for sequential tasks and propose modifications to a network of LIF neurons. We then develop a training scheme to train the proposed SNNs with improved inherent recurrence dynamics.
  arXiv  Detail & Related papers  (2021-09-04T17:13:28Z)
• Experiments on Properties of Hidden Structures of Sparse Neural Networks [1.227734309612871]
  We show how sparsity can be achieved through priors, pruning, and during learning, and answer questions on the relationship between the structure of Neural Networks and their performance. Within our experiments, we show how magnitude class blinded pruning achieves 97.5% on MNIST with 80% compression and re-training, which is 0.5 points more than without compression. Further, performance prediction for Recurrent Networks learning the Reber grammar shows an $R2$ of up to 0.81 given only structural information.
  arXiv  Detail & Related papers  (2021-07-27T16:18:13Z)
• Modeling Token-level Uncertainty to Learn Unknown Concepts in SLU via Calibrated Dirichlet Prior RNN [98.4713940310056]
  One major task of spoken language understanding (SLU) in modern personal assistants is to extract semantic concepts from an utterance. Recent research collected question and answer annotated data to learn what is unknown and should be asked. We incorporate softmax-based slot filling neural architectures to model the sequence uncertainty without question supervision.
  arXiv  Detail & Related papers  (2020-10-16T02:12:30Z)
• RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr [60.07531696857743]
  Fine-tuning the deep convolution neural network(CNN) using a pre-trained model helps transfer knowledge learned from larger datasets to the target task. We propose RIFLE - a strategy that deepens backpropagation in transfer learning settings. RIFLE brings meaningful updates to the weights of deep CNN layers and improves low-level feature learning.
  arXiv  Detail & Related papers  (2020-07-07T11:27:43Z)
• Continual Learning in Recurrent Neural Networks [67.05499844830231]
  We evaluate the effectiveness of continual learning methods for processing sequential data with recurrent neural networks (RNNs) We shed light on the particularities that arise when applying weight-importance methods, such as elastic weight consolidation, to RNNs. We show that the performance of weight-importance methods is not directly affected by the length of the processed sequences, but rather by high working memory requirements.
  arXiv  Detail & Related papers  (2020-06-22T10:05:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.