Self-Regulated Neurogenesis for Online Data-Incremental Learning

• URL: http://arxiv.org/abs/2403.14684v2
• Date: Thu, 26 Jun 2025 11:35:57 GMT
• Title: Self-Regulated Neurogenesis for Online Data-Incremental Learning
• Authors: Murat Onur Yildirim, Elif Ceren Gok Yildirim, Decebal Constantin Mocanu, Joaquin Vanschoren,
• Abstract summary: SERENA encodes each concept in a specialized network path called 'concept cell'<n>Once a concept is learned, its corresponding concept cell is frozen, effectively preventing the forgetting of previously acquired information.<n> Experimental results show that our method not only establishes new state-of-the-art results across ten benchmarks but also remarkably surpasses offline supervised batch learning performance.
• Score: 9.254419196812233
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural networks often struggle with catastrophic forgetting when learning sequences of tasks or data streams, unlike humans who can continuously learn and consolidate new concepts even in the absence of explicit cues. Online data-incremental learning seeks to emulate this capability by processing each sample only once, without having access to task or stream cues at any point in time since this is more realistic compared to offline setups, where all data from novel class(es) is assumed to be readily available. However, existing methods typically rely on storing the subsets of data in memory or expanding the initial model architecture, resulting in significant computational overhead. Drawing inspiration from 'self-regulated neurogenesis'-brain's mechanism for creating specialized regions or circuits for distinct functions-we propose a novel approach SERENA which encodes each concept in a specialized network path called 'concept cell', integrated into a single over-parameterized network. Once a concept is learned, its corresponding concept cell is frozen, effectively preventing the forgetting of previously acquired information. Furthermore, we introduce two new continual learning scenarios that more closely reflect real-world conditions, characterized by gradually changing sample sizes. Experimental results show that our method not only establishes new state-of-the-art results across ten benchmarks but also remarkably surpasses offline supervised batch learning performance. The code is available at https://github.com/muratonuryildirim/serena.

Related papers

• Towards Better De-raining Generalization via Rainy Characteristics Memorization and Replay [74.54047495424618]
  Current image de-raining methods primarily learn from a limited dataset.<n>We introduce a new framework that enables networks to progressively expand their de-raining knowledge base.
  arXiv  Detail & Related papers  (2025-06-03T05:50:00Z)
• A Unified Framework for Neural Computation and Learning Over Time [56.44910327178975]
  Hamiltonian Learning is a novel unified framework for learning with neural networks "over time" It is based on differential equations that: (i) can be integrated without the need of external software solvers; (ii) generalize the well-established notion of gradient-based learning in feed-forward and recurrent networks; (iii) open to novel perspectives.
  arXiv  Detail & Related papers  (2024-09-18T14:57:13Z)
• Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning [40.13416912075668]
  We propose a rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks. Our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order.
  arXiv  Detail & Related papers  (2023-06-21T01:43:25Z)
• IF2Net: Innately Forgetting-Free Networks for Continual Learning [49.57495829364827]
  Continual learning can incrementally absorb new concepts without interfering with previously learned knowledge. Motivated by the characteristics of neural networks, we investigated how to design an Innately Forgetting-Free Network (IF2Net) IF2Net allows a single network to inherently learn unlimited mapping rules without telling task identities at test time.
  arXiv  Detail & Related papers  (2023-06-18T05:26:49Z)
• Dealing with Cross-Task Class Discrimination in Online Continual Learning [54.31411109376545]
  This paper argues for another challenge in class-incremental learning (CIL) How to establish decision boundaries between the classes of the new task and old tasks with no (or limited) access to the old task data. A replay method saves a small amount of data (replay data) from previous tasks. When a batch of current task data arrives, the system jointly trains the new data and some sampled replay data. This paper argues that the replay approach also has a dynamic training bias issue which reduces the effectiveness of the replay data in solving the CTCD problem.
  arXiv  Detail & Related papers  (2023-05-24T02:52:30Z)
• Improving Representational Continuity via Continued Pretraining [76.29171039601948]
  Transfer learning community (LP-FT) outperforms naive training and other continual learning methods. LP-FT also reduces forgetting in a real world satellite remote sensing dataset (FMoW) variant of LP-FT gets state-of-the-art accuracies on an NLP continual learning benchmark.
  arXiv  Detail & Related papers  (2023-02-26T10:39:38Z)
• Continual Prune-and-Select: Class-incremental learning with specialized subnetworks [66.4795381419701]
  Continual-Prune-and-Select (CP&S) is capable of sequentially learning 10 tasks from ImageNet-1000 keeping an accuracy around 94% with negligible forgetting. This is a first-of-its-kind result in class-incremental learning.
  arXiv  Detail & Related papers  (2022-08-09T10:49:40Z)
• Learning Rate Curriculum [75.98230528486401]
  We propose a novel curriculum learning approach termed Learning Rate Curriculum (LeRaC) LeRaC uses a different learning rate for each layer of a neural network to create a data-agnostic curriculum during the initial training epochs. We compare our approach with Curriculum by Smoothing (CBS), a state-of-the-art data-agnostic curriculum learning approach.
  arXiv  Detail & Related papers  (2022-05-18T18:57:36Z)
• APP: Anytime Progressive Pruning [104.36308667437397]
  We propose a novel way of training a neural network with a target sparsity in a particular case of online learning: the anytime learning at macroscale paradigm (ALMA) The proposed approach significantly outperforms the baseline dense and Anytime OSP models across multiple architectures and datasets under short, moderate, and long-sequence training.
  arXiv  Detail & Related papers  (2022-04-04T16:38:55Z)
• Exemplar-free Online Continual Learning [7.800379384628357]
  Continual learning aims to learn new tasks from sequentially available data under the condition that each data is observed only once by the learner. Recent works have made remarkable achievements by storing part of learned task data as exemplars for knowledge replay. We propose a novel exemplar-free method by leveraging nearest-class-mean (NCM) classifier.
  arXiv  Detail & Related papers  (2022-02-11T08:03:22Z)
• Being Friends Instead of Adversaries: Deep Networks Learn from Data Simplified by Other Networks [23.886422706697882]
  A different idea has been recently proposed, named Friendly Training, which consists in altering the input data by adding an automatically estimated perturbation. We revisit and extend this idea inspired by the effectiveness of neural generators in the context of Adversarial Machine Learning. We propose an auxiliary multi-layer network that is responsible of altering the input data to make them easier to be handled by the classifier.
  arXiv  Detail & Related papers  (2021-12-18T16:59:35Z)
• A Meta-Learned Neuron model for Continual Learning [0.0]
  Continual learning is the ability to acquire new knowledge without forgetting the previously learned one. In this work, we replace the standard neuron by a meta-learned neuron model. Our approach can memorize dataset-length sequences of training samples, and its learning capabilities generalize to any domain.
  arXiv  Detail & Related papers  (2021-11-03T23:39:14Z)
• Online Continual Learning Via Candidates Voting [7.704949298975352]
  We introduce an effective and memory-efficient method for online continual learning under class-incremental setting. Our proposed method achieves the best results under different benchmark datasets for online continual learning including CIFAR-10, CIFAR-100 and CORE-50.
  arXiv  Detail & Related papers  (2021-10-17T15:45:32Z)
• Incremental Deep Neural Network Learning using Classification Confidence Thresholding [4.061135251278187]
  Most modern neural networks for classification fail to take into account the concept of the unknown. This paper proposes the Classification Confidence Threshold approach to prime neural networks for incremental learning.
  arXiv  Detail & Related papers  (2021-06-21T22:46:28Z)
• PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning [109.84770951839289]
  We present PredRNN, a new recurrent network for learning visual dynamics from historical context. We show that our approach obtains highly competitive results on three standard datasets.
  arXiv  Detail & Related papers  (2021-03-17T08:28:30Z)
• Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memory [79.42778415729475]
  We propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning. We show how to extend the architecture of a simple RNN by separating its hidden state into different modules. We discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies.
  arXiv  Detail & Related papers  (2020-06-29T08:35:49Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)
• iTAML: An Incremental Task-Agnostic Meta-learning Approach [123.10294801296926]
  Humans can continuously learn new knowledge as their experience grows. Previous learning in deep neural networks can quickly fade out when they are trained on a new task. We introduce a novel meta-learning approach that seeks to maintain an equilibrium between all encountered tasks.
  arXiv  Detail & Related papers  (2020-03-25T21:42:48Z)
• Triple Memory Networks: a Brain-Inspired Method for Continual Learning [35.40452724755021]
  A neural network adjusts its parameters when learning a new task, but then fails to conduct the old tasks well. The brain has a powerful ability to continually learn new experience without catastrophic interference. Inspired by such brain strategy, we propose a novel approach named triple memory networks (TMNs) for continual learning.
  arXiv  Detail & Related papers  (2020-03-06T11:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.