Dataset Knowledge Transfer for Class-Incremental Learning without Memory
- URL: http://arxiv.org/abs/2110.08421v1
- Date: Sat, 16 Oct 2021 00:33:33 GMT
- Title: Dataset Knowledge Transfer for Class-Incremental Learning without Memory
- Authors: Habib Slim, Eden Belouadah, Adrian Popescu and Darian Onchis
- Abstract summary: We tackle class-incremental learning without memory by adapting prediction bias correction.
It is proposed when a memory is allowed and cannot be directly used without memory, since samples of past classes are required.
We introduce a two-step learning process which allows the transfer of bias correction parameters between reference and target datasets.
- Score: 12.569286058146343
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Incremental learning enables artificial agents to learn from sequential data.
While important progress was made by exploiting deep neural networks,
incremental learning remains very challenging. This is particularly the case
when no memory of past data is allowed and catastrophic forgetting has a strong
negative effect. We tackle class-incremental learning without memory by
adapting prediction bias correction, a method which makes predictions of past
and new classes more comparable. It was proposed when a memory is allowed and
cannot be directly used without memory, since samples of past classes are
required. We introduce a two-step learning process which allows the transfer of
bias correction parameters between reference and target datasets. Bias
correction is first optimized offline on reference datasets which have an
associated validation memory. The obtained correction parameters are then
transferred to target datasets, for which no memory is available. The second
contribution is to introduce a finer modeling of bias correction by learning
its parameters per incremental state instead of the usual past vs. new class
modeling. The proposed dataset knowledge transfer is applicable to any
incremental method which works without memory. We test its effectiveness by
applying it to four existing methods. Evaluation with four target datasets and
different configurations shows consistent improvement, with practically no
computational and memory overhead.
Related papers
- Inverted Activations [5.070981175240306]
This paper proposes a modification to the handling of activation tensors in pointwise nonlinearity layers.
Our method involves saving the output tensor instead, reducing the memory required when the subsequent layer also saves its input tensor.
Experimental results confirm that our method significantly reduces memory usage without affecting training accuracy.
arXiv Detail & Related papers (2024-07-22T11:11:17Z) - Adaptive Rentention & Correction for Continual Learning [114.5656325514408]
A common problem in continual learning is the classification layer's bias towards the most recent task.
We name our approach Adaptive Retention & Correction (ARC)
ARC achieves an average performance increase of 2.7% and 2.6% on the CIFAR-100 and Imagenet-R datasets.
arXiv Detail & Related papers (2024-05-23T08:43:09Z) - Enhancing Consistency and Mitigating Bias: A Data Replay Approach for
Incremental Learning [100.7407460674153]
Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks.
To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks.
However, it is not expected in practice considering the memory constraint or data privacy issue.
As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
arXiv Detail & Related papers (2024-01-12T12:51:12Z) - AdaLomo: Low-memory Optimization with Adaptive Learning Rate [59.64965955386855]
We introduce low-memory optimization with adaptive learning rate (AdaLomo) for large language models.
AdaLomo results on par with AdamW, while significantly reducing memory requirements, thereby lowering the hardware barrier to training large language models.
arXiv Detail & Related papers (2023-10-16T09:04:28Z) - Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of
Language Model [92.55145016562867]
We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance.
Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
arXiv Detail & Related papers (2023-05-24T15:52:08Z) - Remind of the Past: Incremental Learning with Analogical Prompts [30.333352182303038]
We design an analogy-making mechanism to remap the new data into the old class by prompt tuning.
It mimics the feature distribution of the target old class on the old model using only samples of new classes.
The learnt prompts are further used to estimate and counteract the representation shift caused by fine-tuning for the historical prototypes.
arXiv Detail & Related papers (2023-03-24T10:18:28Z) - Learning to Unlearn: Instance-wise Unlearning for Pre-trained
Classifiers [71.70205894168039]
We consider instance-wise unlearning, of which the goal is to delete information on a set of instances from a pre-trained model.
We propose two methods that reduce forgetting on the remaining data: 1) utilizing adversarial examples to overcome forgetting at the representation-level and 2) leveraging weight importance metrics to pinpoint network parameters guilty of propagating unwanted information.
arXiv Detail & Related papers (2023-01-27T07:53:50Z) - A Memory Transformer Network for Incremental Learning [64.0410375349852]
We study class-incremental learning, a training setup in which new classes of data are observed over time for the model to learn from.
Despite the straightforward problem formulation, the naive application of classification models to class-incremental learning results in the "catastrophic forgetting" of previously seen classes.
One of the most successful existing methods has been the use of a memory of exemplars, which overcomes the issue of catastrophic forgetting by saving a subset of past data into a memory bank and utilizing it to prevent forgetting when training future tasks.
arXiv Detail & Related papers (2022-10-10T08:27:28Z) - Representation Memorization for Fast Learning New Knowledge without
Forgetting [36.55736909586313]
The ability to quickly learn new knowledge is a big step towards human-level intelligence.
We consider scenarios that require learning new classes or data distributions quickly and incrementally over time.
We propose "Memory-based Hebbian Adaptation" to tackle the two major challenges.
arXiv Detail & Related papers (2021-08-28T07:54:53Z) - ZS-IL: Looking Back on Learned ExperiencesFor Zero-Shot Incremental
Learning [9.530976792843495]
We propose an on-call transfer set to provide past experiences whenever a new class arises in the data stream.
ZS-IL demonstrates significantly better performance on the well-known datasets (CIFAR-10, Tiny-ImageNet) in both Task-IL and Class-IL settings.
arXiv Detail & Related papers (2021-03-22T22:43:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.