Learnability and Algorithm for Continual Learning
- URL: http://arxiv.org/abs/2306.12646v1
- Date: Thu, 22 Jun 2023 03:08:42 GMT
- Title: Learnability and Algorithm for Continual Learning
- Authors: Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Bing Liu
- Abstract summary: Class Incremental Learning (CIL) learns a sequence of tasks consisting of disjoint sets of concepts or classes.
This paper shows that CIL is learnable. Based on the theory, a new CIL algorithm is also proposed.
- Score: 7.7046692574332285
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: This paper studies the challenging continual learning (CL) setting of Class
Incremental Learning (CIL). CIL learns a sequence of tasks consisting of
disjoint sets of concepts or classes. At any time, a single model is built that
can be applied to predict/classify test instances of any classes learned thus
far without providing any task related information for each test instance.
Although many techniques have been proposed for CIL, they are mostly empirical.
It has been shown recently that a strong CIL system needs a strong within-task
prediction (WP) and a strong out-of-distribution (OOD) detection for each task.
However, it is still not known whether CIL is actually learnable. This paper
shows that CIL is learnable. Based on the theory, a new CIL algorithm is also
proposed. Experimental results demonstrate its effectiveness.
Related papers
- Class Incremental Learning via Likelihood Ratio Based Task Prediction [20.145128455767587]
An emerging theory-guided approach is to train a task-specific model for each task in a shared network for all tasks.
This paper argues that using a traditional OOD detector for task-id prediction is sub-optimal because additional information can be exploited.
We call the new method TPL (Task-id Prediction based on Likelihood Ratio)
It markedly outperforms strong CIL baselines and has negligible catastrophic forgetting.
arXiv Detail & Related papers (2023-09-26T16:25:57Z) - Multiclass Boosting: Simple and Intuitive Weak Learning Criteria [72.71096438538254]
We give a simple and efficient boosting algorithm, that does not require realizability assumptions.
We present a new result on boosting for list learners, as well as provide a novel proof for the characterization of multiclass PAC learning.
arXiv Detail & Related papers (2023-07-02T19:26:58Z) - Class-Incremental Learning: A Survey [84.30083092434938]
Class-Incremental Learning (CIL) enables the learner to incorporate the knowledge of new classes incrementally.
CIL tends to catastrophically forget the characteristics of former ones, and its performance drastically degrades.
We provide a rigorous and unified evaluation of 17 methods in benchmark image classification tasks to find out the characteristics of different algorithms.
arXiv Detail & Related papers (2023-02-07T17:59:05Z) - NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision
Research [96.53307645791179]
We introduce the Never-Ending VIsual-classification Stream (NEVIS'22), a benchmark consisting of a stream of over 100 visual classification tasks.
Despite being limited to classification, the resulting stream has a rich diversity of tasks from OCR, to texture analysis, scene recognition, and so forth.
Overall, NEVIS'22 poses an unprecedented challenge for current sequential learning approaches due to the scale and diversity of tasks.
arXiv Detail & Related papers (2022-11-15T18:57:46Z) - A Theoretical Study on Solving Continual Learning [13.186315474669287]
This study shows that the CIL problem can be decomposed into two sub-problems: Within-task Prediction (WP) and Task-id Prediction (TP)
It further proves that TP is correlated with out-of-distribution (OOD) detection, which connects CIL and OOD detection.
The key conclusion of this study is that regardless of whether WP and TP or OOD detection are defined explicitly or implicitly by a CIL algorithm, good WP and good TP or OOD detection are necessary and sufficient for good CIL performances.
arXiv Detail & Related papers (2022-11-04T17:45:55Z) - Chaos is a Ladder: A New Theoretical Understanding of Contrastive
Learning via Augmentation Overlap [64.60460828425502]
We propose a new guarantee on the downstream performance of contrastive learning.
Our new theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations.
We propose an unsupervised model selection metric ARC that aligns well with downstream accuracy.
arXiv Detail & Related papers (2022-03-25T05:36:26Z) - Continual Learning Based on OOD Detection and Task Masking [7.7046692574332285]
This paper proposes a novel unified approach based on out-of-distribution (OOD) detection and task masking, called CLOM, to solve both problems.
Our evaluation shows that CLOM outperforms existing state-of-the-art baselines by large margins.
arXiv Detail & Related papers (2022-03-17T17:10:12Z) - vCLIMB: A Novel Video Class Incremental Learning Benchmark [53.90485760679411]
We introduce vCLIMB, a novel video continual learning benchmark.
vCLIMB is a standardized test-bed to analyze catastrophic forgetting of deep models in video continual learning.
We propose a temporal consistency regularization that can be applied on top of memory-based continual learning methods.
arXiv Detail & Related papers (2022-01-23T22:14:17Z) - CLASSIC: Continual and Contrastive Learning of Aspect Sentiment
Classification Tasks [23.515930312505954]
This paper studies continual learning of a sequence of aspect sentiment classification(ASC) tasks in a particular CL setting called domain incremental learning (DIL)
The DIL setting is particularly suited to ASC because in testing the system needs not know the task/domain to which the test data belongs.
The key novelty is a contrastive continual learning method that enables both knowledge transfer across tasks and knowledge distillation from old tasks to the new task.
arXiv Detail & Related papers (2021-12-05T23:55:53Z) - ClaRe: Practical Class Incremental Learning By Remembering Previous
Class Representations [9.530976792843495]
Class Incremental Learning (CIL) tends to learn new concepts perfectly, but not at the expense of performance and accuracy for old data.
ClaRe is an efficient solution for CIL by remembering the representations of learned classes in each increment.
ClaRe has a better generalization than prior methods thanks to producing diverse instances from the distribution of previously learned classes.
arXiv Detail & Related papers (2021-03-29T10:39:42Z) - Rethinking Few-Shot Image Classification: a Good Embedding Is All You
Need? [72.00712736992618]
We show that a simple baseline: learning a supervised or self-supervised representation on the meta-training set, outperforms state-of-the-art few-shot learning methods.
An additional boost can be achieved through the use of self-distillation.
We believe that our findings motivate a rethinking of few-shot image classification benchmarks and the associated role of meta-learning algorithms.
arXiv Detail & Related papers (2020-03-25T17:58:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.