Knowledge Capture and Replay for Continual Learning

• URL: http://arxiv.org/abs/2012.06789v2
• Date: Thu, 29 Apr 2021 14:17:52 GMT
• Title: Knowledge Capture and Replay for Continual Learning
• Authors: Saisubramaniam Gopalakrishnan, Pranshu Ranjan Singh, Haytham Fayek, Savitha Ramasamy, Arulmurugan Ambikapathi
• Abstract summary: We introduce em flashcards, which are visual representations that em capture encoded knowledge of a network. In a continual learning scenario, flashcards help to prevent forgetting and consolidating knowledge of all the previous tasks. We demonstrate the efficacy of flashcards in capturing learned knowledge representation (as an alternative to the original dataset) and empirically validate on a variety of continual learning tasks.
• Score: 0.4980584790669266
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks have shown promise in several domains, and the learned data (task) specific information is implicitly stored in the network parameters. Extraction and utilization of encoded knowledge representations are vital when data is no longer available in the future, especially in a continual learning scenario. In this work, we introduce {\em flashcards}, which are visual representations that {\em capture} the encoded knowledge of a network as a recursive function of predefined random image patterns. In a continual learning scenario, flashcards help to prevent catastrophic forgetting and consolidating knowledge of all the previous tasks. Flashcards need to be constructed only before learning the subsequent task, and hence, independent of the number of tasks trained before. We demonstrate the efficacy of flashcards in capturing learned knowledge representation (as an alternative to the original dataset) and empirically validate on a variety of continual learning tasks: reconstruction, denoising, task-incremental learning, and new-instance learning classification, using several heterogeneous benchmark datasets. Experimental evidence indicates that: (i) flashcards as a replay strategy is { \em task agnostic}, (ii) performs better than generative replay, and (iii) is on par with episodic replay without additional memory overhead.

Related papers

• Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning [70.64617500380287]
  Continual learning allows models to learn from new data while retaining previously learned knowledge. The semantic knowledge available in the label information of the images, offers important semantic information that can be related with previously acquired knowledge of semantic classes. We propose integrating semantic guidance within and across tasks by capturing semantic similarity using text embeddings.
  arXiv  Detail & Related papers  (2024-08-02T07:51:44Z)
• Look At Me, No Replay! SurpriseNet: Anomaly Detection Inspired Class Incremental Learning [14.529164755845688]
  Continual learning aims to create artificial neural networks capable of accumulating knowledge and skills through incremental training on a sequence of tasks. The main challenge of continual learning is catastrophic interference, wherein new knowledge overrides or interferes with past knowledge, leading to forgetting. A proposed solution, SurpriseNet, addresses catastrophic interference by employing a parameter isolation method and learning cross-task knowledge using an auto-encoder inspired by anomaly detection.
  arXiv  Detail & Related papers  (2023-10-30T22:16:26Z)
• Measures of Information Reflect Memorization Patterns [53.71420125627608]
  We show that the diversity in the activation patterns of different neurons is reflective of model generalization and memorization. Importantly, we discover that information organization points to the two forms of memorization, even for neural activations computed on unlabelled in-distribution examples.
  arXiv  Detail & Related papers  (2022-10-17T20:15:24Z)
• Learning with Recoverable Forgetting [77.56338597012927]
  Learning wIth Recoverable Forgetting explicitly handles the task- or sample-specific knowledge removal and recovery. Specifically, LIRF brings in two innovative schemes, namely knowledge deposit and withdrawal. We conduct experiments on several datasets, and demonstrate that the proposed LIRF strategy yields encouraging results with gratifying generalization capability.
  arXiv  Detail & Related papers  (2022-07-17T16:42:31Z)
• Memory-Based Label-Text Tuning for Few-Shot Class-Incremental Learning [20.87638654650383]
  We propose leveraging the label-text information by adopting the memory prompt. The memory prompt can learn new data sequentially, and meanwhile store the previous knowledge. Experiments show that our proposed method outperforms all prior state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-07-03T13:15:45Z)
• Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning [113.58691755215663]
  We develop RetroPrompt to help a model strike a balance between generalization and memorization. In contrast with vanilla prompt learning, RetroPrompt constructs an open-book knowledge-store from training instances. Extensive experiments demonstrate that RetroPrompt can obtain better performance in both few-shot and zero-shot settings.
  arXiv  Detail & Related papers  (2022-05-29T16:07:30Z)
• Continual Learning via Bit-Level Information Preserving [88.32450740325005]
  We study the continual learning process through the lens of information theory. We propose Bit-Level Information Preserving (BLIP) that preserves the information gain on model parameters. BLIP achieves close to zero forgetting while only requiring constant memory overheads throughout continual learning.
  arXiv  Detail & Related papers  (2021-05-10T15:09:01Z)
• MetaKernel: Learning Variational Random Features with Limited Labels [120.90737681252594]
  Few-shot learning deals with the fundamental and challenging problem of learning from a few annotated samples, while being able to generalize well on new tasks. We propose meta-learning kernels with random Fourier features for few-shot learning, we call Meta Kernel.
  arXiv  Detail & Related papers  (2021-05-08T21:24:09Z)
• Learning to Continually Learn Rapidly from Few and Noisy Data [19.09933805011466]
  Continual learning could be achieved via replay -- by concurrently training externally stored old data while learning a new task. By employing a meta-learner, which textitlearns a learning rate per parameter per past task, we found that base learners produced strong results when less memory was available.
  arXiv  Detail & Related papers  (2021-03-06T08:29:47Z)
• Meta-Learning with Sparse Experience Replay for Lifelong Language Learning [26.296412053816233]
  We propose a novel approach to lifelong learning of language tasks based on meta-learning with sparse experience replay. We show that under the realistic setting of performing a single pass on a stream of tasks, our method obtains state-of-the-art results on lifelong text classification and relation extraction.
  arXiv  Detail & Related papers  (2020-09-10T14:36:38Z)
• Using Hindsight to Anchor Past Knowledge in Continual Learning [36.271906785418864]
  In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. In this work, we call anchoring, where the learner uses bilevel optimization to update its knowledge on the current task, while keeping intact the predictions on past tasks.
  arXiv  Detail & Related papers  (2020-02-19T13:21:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.