Infinite dSprites for Disentangled Continual Learning: Separating Memory Edits from Generalization

• URL: http://arxiv.org/abs/2312.16731v3
• Date: Mon, 29 Jul 2024 21:32:01 GMT
• Title: Infinite dSprites for Disentangled Continual Learning: Separating Memory Edits from Generalization
• Authors: Sebastian Dziadzio, Çağatay Yıldız, Gido M. van de Ven, Tomasz Trzciński, Tinne Tuytelaars, Matthias Bethge,
• Abstract summary: We introduce Infinite dSprites, a parsimonious tool for creating continual classification benchmarks of arbitrary length. We show that over a sufficiently long time horizon, the performance of all major types of continual learning methods deteriorates on this simple benchmark. In a simple setting with direct supervision on the generative factors, we show how learning class-agnostic transformations offers a way to circumvent catastrophic forgetting.
• Score: 36.23065731463065
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The ability of machine learning systems to learn continually is hindered by catastrophic forgetting, the tendency of neural networks to overwrite previously acquired knowledge when learning a new task. Existing methods mitigate this problem through regularization, parameter isolation, or rehearsal, but they are typically evaluated on benchmarks comprising only a handful of tasks. In contrast, humans are able to learn over long time horizons in dynamic, open-world environments, effortlessly memorizing unfamiliar objects and reliably recognizing them under various transformations. To make progress towards closing this gap, we introduce Infinite dSprites, a parsimonious tool for creating continual classification and disentanglement benchmarks of arbitrary length and with full control over generative factors. We show that over a sufficiently long time horizon, the performance of all major types of continual learning methods deteriorates on this simple benchmark. This result highlights an important and previously overlooked aspect of continual learning: given a finite modelling capacity and an arbitrarily long learning horizon, efficient learning requires memorizing class-specific information and accumulating knowledge about general mechanisms. In a simple setting with direct supervision on the generative factors, we show how learning class-agnostic transformations offers a way to circumvent catastrophic forgetting and improve classification accuracy over time. Our approach sets the stage for continual learning over hundreds of tasks with explicit control over memorization and forgetting, emphasizing open-set classification and one-shot generalization.

Related papers

• Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
  Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information. We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting. Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
  arXiv  Detail & Related papers  (2023-09-15T17:10:51Z)
• Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning [40.13416912075668]
  We propose a rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks. Our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order.
  arXiv  Detail & Related papers  (2023-06-21T01:43:25Z)
• Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning [54.7584721943286]
  Continual Learning (CL) methods aim to enable machine learning models to learn new tasks without catastrophic forgetting of those that have been previously mastered. Existing CL approaches often keep a buffer of previously-seen samples, perform knowledge distillation, or use regularization techniques towards this goal. We propose to only activate and select sparse neurons for learning current and past tasks at any stage.
  arXiv  Detail & Related papers  (2022-02-21T13:25:03Z)
• Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System [13.041607703862724]
  We propose CLS-ER, a novel dual memory experience replay (ER) method. New knowledge is acquired while aligning the decision boundaries with the semantic memories. Our approach achieves state-of-the-art performance on standard benchmarks.
  arXiv  Detail & Related papers  (2022-01-29T15:15:23Z)
• Continually Learning Self-Supervised Representations with Projected Functional Regularization [39.92600544186844]
  Recent self-supervised learning methods are able to learn high-quality image representations and are closing the gap with supervised methods. These methods are unable to acquire new knowledge incrementally -- they are, in fact, mostly used only as a pre-training phase with IID data. To prevent forgetting of previous knowledge, we propose the usage of functional regularization.
  arXiv  Detail & Related papers  (2021-12-30T11:59:23Z)
• Center Loss Regularization for Continual Learning [0.0]
  In general, neural networks lack the ability to learn different tasks sequentially. Our approach remembers old tasks by projecting the representations of new tasks close to that of old tasks. We demonstrate that our approach is scalable, effective, and gives competitive performance compared to state-of-the-art continual learning methods.
  arXiv  Detail & Related papers  (2021-10-21T17:46:44Z)
• Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors [63.21036904487014]
  Continual learning of new knowledge over time is one desirable capability for intelligent systems to recognize more and more classes of objects. We propose a simple yet effective fusion mechanism by including all the previously learned feature extractors into the intelligent model. Experiments on multiple classification tasks show that the proposed approach can effectively reduce the forgetting of old knowledge, achieving state-of-the-art continual learning performance.
  arXiv  Detail & Related papers  (2021-04-28T07:49:24Z)
• Essentials for Class Incremental Learning [43.306374557919646]
  Class-incremental learning results on CIFAR-100 and ImageNet improve over the state-of-the-art by a large margin, while keeping the approach simple.
  arXiv  Detail & Related papers  (2021-02-18T18:01:06Z)
• Continual Learning in Low-rank Orthogonal Subspaces [86.36417214618575]
  In continual learning (CL), a learner is faced with a sequence of tasks, arriving one after the other, and the goal is to remember all the tasks once the learning experience is finished. The prior art in CL uses episodic memory, parameter regularization or network structures to reduce interference among tasks, but in the end, all the approaches learn different tasks in a joint vector space. We propose to learn tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference.
  arXiv  Detail & Related papers  (2020-10-22T12:07:43Z)
• Bilevel Continual Learning [76.50127663309604]
  We present a novel framework of continual learning named "Bilevel Continual Learning" (BCL) Our experiments on continual learning benchmarks demonstrate the efficacy of the proposed BCL compared to many state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-30T16:00:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.