Continual Learning with Columnar Spiking Neural Networks

• URL: http://arxiv.org/abs/2506.17169v1
• Date: Fri, 20 Jun 2025 17:13:38 GMT
• Title: Continual Learning with Columnar Spiking Neural Networks
• Authors: Denis Larionov, Nikolay Bazenkov, Mikhail Kiselev,
• Abstract summary: We show that microcolumns adapt most efficiently to new tasks when they lack shared structure with prior learning.<n>Our optimal configuration learns ten sequential MNIST tasks effectively, maintaining 92% accuracy on each.<n>It shows low forgetting, with only 4% performance degradation on the first task after training on nine subsequent tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study investigates columnar-organized spiking neural networks (SNNs) for continual learning and catastrophic forgetting. Using CoLaNET (Columnar Layered Network), we show that microcolumns adapt most efficiently to new tasks when they lack shared structure with prior learning. We demonstrate how CoLaNET hyperparameters govern the trade-off between retaining old knowledge (stability) and acquiring new information (plasticity). Our optimal configuration learns ten sequential MNIST tasks effectively, maintaining 92% accuracy on each. It shows low forgetting, with only 4% performance degradation on the first task after training on nine subsequent tasks.

Related papers

• Learning without Isolation: Pathway Protection for Continual Learning [64.3476595369537]
  Deep networks are prone to catastrophic forgetting during sequential task learning.<n>We propose a novel CL framework, learning without isolation(LwI), where model fusion is formulated as graph matching.<n>Thanks to the sparsity of activation channels in a deep network, LwI can adaptively allocate available pathways for a new task, realizing pathway protection and addressing catastrophic forgetting.
  arXiv  Detail & Related papers  (2025-05-24T07:16:55Z)
• Eidetic Learning: an Efficient and Provable Solution to Catastrophic Forgetting [13.412573082645096]
  We present a method, Eidetic Learning, that provably solves catastrophic forgetting.<n>A network trained with Eidetic Learning -- here, an EideticNet -- requires no rehearsal or replay.<n>An EideticNet is easy to implement and train, is efficient, and has time and space complexity linear in the number of parameters.
  arXiv  Detail & Related papers  (2025-02-13T17:10:43Z)
• ALADE-SNN: Adaptive Logit Alignment in Dynamically Expandable Spiking Neural Networks for Class Incremental Learning [15.022211557367273]
  We develop spiking neural networks (SNNs) with dynamic structures for Class Incremental Learning (CIL)<n>We propose the ALADE-SNN framework, which includes adaptive logit alignment for balanced feature representation and OtoN suppression to manage weights mapping frozen old features to new classes during training.<n>Experiment results show that ALADE-SNN achieves an average incremental accuracy of 75.42 on the CIFAR100-B0 benchmark over 10 incremental steps.
  arXiv  Detail & Related papers  (2024-12-17T09:13:22Z)
• Order parameters and phase transitions of continual learning in deep neural networks [6.349503549199403]
  Continual learning (CL) enables animals to learn new tasks without erasing prior knowledge.<n> CL in artificial neural networks (NNs) is challenging due to catastrophic forgetting, where new learning degrades performance on older tasks.<n>We present a statistical-mechanics theory of CL in deep, wide NNs, which characterizes the network's input-output mapping as it learns a sequence of tasks.
  arXiv  Detail & Related papers  (2024-07-14T20:22:36Z)
• Towards evolution of Deep Neural Networks through contrastive Self-Supervised learning [0.49157446832511503]
  We propose a framework that is able to evolve deep neural networks using self-supervised learning. Our results show that it is possible to evolve adequate neural networks while reducing the reliance on labelled data.
  arXiv  Detail & Related papers  (2024-06-20T17:38:16Z)
• Compressed Latent Replays for Lightweight Continual Learning on Spiking Neural Networks [45.53312220335389]
  We introduce the first memory-efficient implementation of Latent Replay (LR)-based CL for Spiking Neural Networks (SNNs) LRs combine new samples with latent representations of previously learned data, to mitigate forgetting. Experiments on the Heidelberg SHD dataset with Sample and Class-Incremental tasks reach a Top-1 accuracy of 92.5% and 92%, respectively.
  arXiv  Detail & Related papers  (2024-05-08T09:03:17Z)
• Negotiated Representations to Prevent Forgetting in Machine Learning Applications [0.0]
  Catastrophic forgetting is a significant challenge in the field of machine learning. We propose a novel method for preventing catastrophic forgetting in machine learning applications.
  arXiv  Detail & Related papers  (2023-11-30T22:43:50Z)
• CLR: Channel-wise Lightweight Reprogramming for Continual Learning [63.94773340278971]
  Continual learning aims to emulate the human ability to continually accumulate knowledge over sequential tasks. The main challenge is to maintain performance on previously learned tasks after learning new tasks. We propose a Channel-wise Lightweight Reprogramming approach that helps convolutional neural networks overcome catastrophic forgetting.
  arXiv  Detail & Related papers  (2023-07-21T06:56:21Z)
• Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks [69.38572074372392]
  We present the first results proving that feature learning occurs during training with a nonlinear model on multiple tasks. Our key insight is that multi-task pretraining induces a pseudo-contrastive loss that favors representations that align points that typically have the same label across tasks.
  arXiv  Detail & Related papers  (2023-07-13T16:39:08Z)
• IF2Net: Innately Forgetting-Free Networks for Continual Learning [49.57495829364827]
  Continual learning can incrementally absorb new concepts without interfering with previously learned knowledge. Motivated by the characteristics of neural networks, we investigated how to design an Innately Forgetting-Free Network (IF2Net) IF2Net allows a single network to inherently learn unlimited mapping rules without telling task identities at test time.
  arXiv  Detail & Related papers  (2023-06-18T05:26:49Z)
• Continual Prune-and-Select: Class-incremental learning with specialized subnetworks [66.4795381419701]
  Continual-Prune-and-Select (CP&S) is capable of sequentially learning 10 tasks from ImageNet-1000 keeping an accuracy around 94% with negligible forgetting. This is a first-of-its-kind result in class-incremental learning.
  arXiv  Detail & Related papers  (2022-08-09T10:49:40Z)
• Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning [54.7584721943286]
  Continual Learning (CL) methods aim to enable machine learning models to learn new tasks without catastrophic forgetting of those that have been previously mastered. Existing CL approaches often keep a buffer of previously-seen samples, perform knowledge distillation, or use regularization techniques towards this goal. We propose to only activate and select sparse neurons for learning current and past tasks at any stage.
  arXiv  Detail & Related papers  (2022-02-21T13:25:03Z)
• GROWN: GRow Only When Necessary for Continual Learning [39.56829374809613]
  Catastrophic forgetting is a notorious issue in deep learning, referring to the fact that Deep Neural Networks (DNN) could forget the knowledge about earlier tasks when learning new tasks. To address this issue, continual learning has been developed to learn new tasks sequentially and perform knowledge transfer from the old tasks to the new ones without forgetting. GROWN is a novel end-to-end continual learning framework to dynamically grow the model only when necessary.
  arXiv  Detail & Related papers  (2021-10-03T02:31:04Z)
• Self-supervised Knowledge Distillation for Few-shot Learning [123.10294801296926]
  Few shot learning is a promising learning paradigm due to its ability to learn out of order distributions quickly with only a few samples. We propose a simple approach to improve the representation capacity of deep neural networks for few-shot learning tasks. Our experiments show that, even in the first stage, self-supervision can outperform current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-17T11:27:00Z)
• Continuous Learning in a Single-Incremental-Task Scenario with Spike Features [0.0]
  Deep Neural Networks (DNNs) have two key deficiencies, their dependence on high precision computing and their inability to perform sequential learning. Here, we use bio-inspired Spike Timing Dependent Plasticity (STDP)in the feature extraction layers of the network with instantaneous neurons to extract meaningful features. In the classification sections of the network we use a modified synaptic intelligence that we refer to as cost per synapse metric as a regularizer to immunize the network against catastrophic forgetting.
  arXiv  Detail & Related papers  (2020-05-03T16:18:20Z)
• iTAML: An Incremental Task-Agnostic Meta-learning Approach [123.10294801296926]
  Humans can continuously learn new knowledge as their experience grows. Previous learning in deep neural networks can quickly fade out when they are trained on a new task. We introduce a novel meta-learning approach that seeks to maintain an equilibrium between all encountered tasks.
  arXiv  Detail & Related papers  (2020-03-25T21:42:48Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.