Saliency Guided Experience Packing for Replay in Continual Learning
- URL: http://arxiv.org/abs/2109.04954v1
- Date: Fri, 10 Sep 2021 15:54:58 GMT
- Title: Saliency Guided Experience Packing for Replay in Continual Learning
- Authors: Gobinda Saha and Kaushik Roy
- Abstract summary: We propose a new approach for experience replay, where we select the past experiences by looking at the saliency maps.
While learning a new task, we replay these memory patches with appropriate zero-padding to remind the model about its past decisions.
- Score: 6.417011237981518
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Artificial learning systems aspire to mimic human intelligence by continually
learning from a stream of tasks without forgetting past knowledge. One way to
enable such learning is to store past experiences in the form of input examples
in episodic memory and replay them when learning new tasks. However,
performance of such method suffers as the size of the memory becomes smaller.
In this paper, we propose a new approach for experience replay, where we select
the past experiences by looking at the saliency maps which provide visual
explanations for the model's decision. Guided by these saliency maps, we pack
the memory with only the parts or patches of the input images important for the
model's prediction. While learning a new task, we replay these memory patches
with appropriate zero-padding to remind the model about its past decisions. We
evaluate our algorithm on diverse image classification datasets and report
better performance than the state-of-the-art approaches. With qualitative and
quantitative analyses we show that our method captures richer summary of past
experiences without any memory increase, and hence performs well with small
episodic memory.
Related papers
- TEAL: New Selection Strategy for Small Buffers in Experience Replay Class Incremental Learning [7.627299398469962]
We introduce TEAL, a novel approach to populate the memory with exemplars.
We show that TEAL improves the average accuracy of the SOTA method XDER as well as ER and ER-ACE on several image recognition benchmarks.
arXiv Detail & Related papers (2024-06-30T12:09:08Z) - What do larger image classifiers memorise? [64.01325988398838]
We show that training examples exhibit an unexpectedly diverse set of memorisation trajectories across model sizes.
We find that knowledge distillation, an effective and popular model compression technique, tends to inhibit memorisation, while also improving generalisation.
arXiv Detail & Related papers (2023-10-09T01:52:07Z) - Continual Learning with Strong Experience Replay [32.154995019080594]
We propose a CL method with Strong Experience Replay (SER)
SER utilizes future experiences mimicked on the current training data, besides distilling past experience from the memory buffer.
Experimental results on multiple image classification datasets show that our SER method surpasses the state-of-the-art methods by a noticeable margin.
arXiv Detail & Related papers (2023-05-23T02:42:54Z) - Improving Image Recognition by Retrieving from Web-Scale Image-Text Data [68.63453336523318]
We introduce an attention-based memory module, which learns the importance of each retrieved example from the memory.
Compared to existing approaches, our method removes the influence of the irrelevant retrieved examples, and retains those that are beneficial to the input query.
We show that it achieves state-of-the-art accuracies in ImageNet-LT, Places-LT and Webvision datasets.
arXiv Detail & Related papers (2023-04-11T12:12:05Z) - Saliency-Augmented Memory Completion for Continual Learning [8.243137410556495]
How to forget is a problem continual learning must address.
Our paper proposes a new saliency-augmented memory completion framework for continual learning.
arXiv Detail & Related papers (2022-12-26T18:06:39Z) - A Memory Transformer Network for Incremental Learning [64.0410375349852]
We study class-incremental learning, a training setup in which new classes of data are observed over time for the model to learn from.
Despite the straightforward problem formulation, the naive application of classification models to class-incremental learning results in the "catastrophic forgetting" of previously seen classes.
One of the most successful existing methods has been the use of a memory of exemplars, which overcomes the issue of catastrophic forgetting by saving a subset of past data into a memory bank and utilizing it to prevent forgetting when training future tasks.
arXiv Detail & Related papers (2022-10-10T08:27:28Z) - Learning to Learn Variational Semantic Memory [132.39737669936125]
We introduce variational semantic memory into meta-learning to acquire long-term knowledge for few-shot learning.
The semantic memory is grown from scratch and gradually consolidated by absorbing information from tasks it experiences.
We formulate memory recall as the variational inference of a latent memory variable from addressed contents.
arXiv Detail & Related papers (2020-10-20T15:05:26Z) - Memory-augmented Dense Predictive Coding for Video Representation
Learning [103.69904379356413]
We propose a new architecture and learning framework Memory-augmented Predictive Coding (MemDPC) for the task.
We investigate visual-only self-supervised video representation learning from RGB frames, or from unsupervised optical flow, or both.
In all cases, we demonstrate state-of-the-art or comparable performance over other approaches with orders of magnitude fewer training data.
arXiv Detail & Related papers (2020-08-03T17:57:01Z) - PeTra: A Sparsely Supervised Memory Model for People Tracking [50.98911178059019]
We propose PeTra, a memory-augmented neural network designed to track entities in its memory slots.
We empirically compare key modeling choices, finding that we can simplify several aspects of the design of the memory module while retaining strong performance.
PeTra is highly effective in both evaluations, demonstrating its ability to track people in its memory despite being trained with limited annotation.
arXiv Detail & Related papers (2020-05-06T17:45:35Z) - Using Hindsight to Anchor Past Knowledge in Continual Learning [36.271906785418864]
In continual learning, the learner faces a stream of data whose distribution changes over time.
Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge.
In this work, we call anchoring, where the learner uses bilevel optimization to update its knowledge on the current task, while keeping intact the predictions on past tasks.
arXiv Detail & Related papers (2020-02-19T13:21:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.