Semantic Drift Compensation for Class-Incremental Learning
- URL: http://arxiv.org/abs/2004.00440v1
- Date: Wed, 1 Apr 2020 13:31:19 GMT
- Title: Semantic Drift Compensation for Class-Incremental Learning
- Authors: Lu Yu, Bart{\l}omiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang,
Yongmei Cheng, Shangling Jui, Joost van de Weijer
- Abstract summary: Class-incremental learning of deep networks sequentially increases the number of classes to be classified.
We propose a new method to estimate the drift, called semantic drift, of features and compensate for it without the need of any exemplars.
- Score: 48.749630494026086
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Class-incremental learning of deep networks sequentially increases the number
of classes to be classified. During training, the network has only access to
data of one task at a time, where each task contains several classes. In this
setting, networks suffer from catastrophic forgetting which refers to the
drastic drop in performance on previous tasks. The vast majority of methods
have studied this scenario for classification networks, where for each new task
the classification layer of the network must be augmented with additional
weights to make room for the newly added classes. Embedding networks have the
advantage that new classes can be naturally included into the network without
adding new weights. Therefore, we study incremental learning for embedding
networks. In addition, we propose a new method to estimate the drift, called
semantic drift, of features and compensate for it without the need of any
exemplars. We approximate the drift of previous tasks based on the drift that
is experienced by current task data. We perform experiments on fine-grained
datasets, CIFAR100 and ImageNet-Subset. We demonstrate that embedding networks
suffer significantly less from catastrophic forgetting. We outperform existing
methods which do not require exemplars and obtain competitive results compared
to methods which store exemplars. Furthermore, we show that our proposed SDC
when combined with existing methods to prevent forgetting consistently improves
results.
Related papers
- Negotiated Representations to Prevent Forgetting in Machine Learning
Applications [0.0]
Catastrophic forgetting is a significant challenge in the field of machine learning.
We propose a novel method for preventing catastrophic forgetting in machine learning applications.
arXiv Detail & Related papers (2023-11-30T22:43:50Z) - FDCNet: Feature Drift Compensation Network for Class-Incremental Weakly
Supervised Object Localization [10.08410402383604]
This work addresses the task of class-incremental weakly supervised object localization (CI-WSOL)
The goal is to incrementally learn object localization for novel classes using only image-level annotations while retaining the ability to localize previously learned classes.
We first present a strong baseline method for CI-WSOL by adapting the strategies of class-incremental classifiers to catastrophic forgetting.
We then propose the feature drift compensation network to compensate for the effects of feature drifts on class scores and localization maps.
arXiv Detail & Related papers (2023-09-17T01:10:45Z) - Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks [69.38572074372392]
We present the first results proving that feature learning occurs during training with a nonlinear model on multiple tasks.
Our key insight is that multi-task pretraining induces a pseudo-contrastive loss that favors representations that align points that typically have the same label across tasks.
arXiv Detail & Related papers (2023-07-13T16:39:08Z) - On the Soft-Subnetwork for Few-shot Class Incremental Learning [67.0373924836107]
We propose a few-shot class incremental learning (FSCIL) method referred to as emphSoft-SubNetworks (SoftNet).
Our objective is to learn a sequence of sessions incrementally, where each session only includes a few training instances per class while preserving the knowledge of the previously learned ones.
We provide comprehensive empirical validations demonstrating that our SoftNet effectively tackles the few-shot incremental learning problem by surpassing the performance of state-of-the-art baselines over benchmark datasets.
arXiv Detail & Related papers (2022-09-15T04:54:02Z) - A Multi-Head Model for Continual Learning via Out-of-Distribution Replay [16.189891444511755]
Many approaches have been proposed to deal with catastrophic forgetting (CF) in continual learning (CL)
This paper proposes an entirely different approach that builds a separate classifier (head) for each task (called a multi-head model) using a transformer network, called MORE.
arXiv Detail & Related papers (2022-08-20T19:17:12Z) - Incremental Embedding Learning via Zero-Shot Translation [65.94349068508863]
Current state-of-the-art incremental learning methods tackle catastrophic forgetting problem in traditional classification networks.
We propose a novel class-incremental method for embedding network, named as zero-shot translation class-incremental method (ZSTCI)
In addition, ZSTCI can easily be combined with existing regularization-based incremental learning methods to further improve performance of embedding networks.
arXiv Detail & Related papers (2020-12-31T08:21:37Z) - Mixed-Privacy Forgetting in Deep Networks [114.3840147070712]
We show that the influence of a subset of the training samples can be removed from the weights of a network trained on large-scale image classification tasks.
Inspired by real-world applications of forgetting techniques, we introduce a novel notion of forgetting in mixed-privacy setting.
We show that our method allows forgetting without having to trade off the model accuracy.
arXiv Detail & Related papers (2020-12-24T19:34:56Z) - On the Exploration of Incremental Learning for Fine-grained Image
Retrieval [45.48333682748607]
We consider the problem of fine-grained image retrieval in an incremental setting, when new categories are added over time.
We propose an incremental learning method to mitigate retrieval performance degradation caused by the forgetting issue.
Our method effectively mitigates the catastrophic forgetting on the original classes while achieving high performance on the new classes.
arXiv Detail & Related papers (2020-10-15T21:07:44Z) - Side-Tuning: A Baseline for Network Adaptation via Additive Side
Networks [95.51368472949308]
Adaptation can be useful in cases when training data is scarce, or when one wishes to encode priors in the network.
In this paper, we propose a straightforward alternative: side-tuning.
arXiv Detail & Related papers (2019-12-31T18:52:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.