Bilevel Continual Learning

• URL: http://arxiv.org/abs/2011.01168v1
• Date: Mon, 2 Nov 2020 18:06:42 GMT
• Title: Bilevel Continual Learning
• Authors: Ammar Shaker, Francesco Alesiani, Shujian Yu, Wenzhe Yin
• Abstract summary: This paper presents Bilevel Continual Learning (BiCL), a general framework for continual learning. BiCL fuses bilevel optimization and recent advances in meta-learning for deep neural networks. Experimental results show BiCL provides competitive performance in terms of accuracy for the current task.
• Score: 18.293397644865454
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual learning (CL) studies the problem of learning a sequence of tasks, one at a time, such that the learning of each new task does not lead to the deterioration in performance on the previously seen ones while exploiting previously learned features. This paper presents Bilevel Continual Learning (BiCL), a general framework for continual learning that fuses bilevel optimization and recent advances in meta-learning for deep neural networks. BiCL is able to train both deep discriminative and generative models under the conservative setting of the online continual learning. Experimental results show that BiCL provides competitive performance in terms of accuracy for the current task while reducing the effect of catastrophic forgetting. This is a concurrent work with [1]. We submitted it to AAAI 2020 and IJCAI 2020. Now we put it on the arxiv for record. Different from [1], we also consider continual generative model as well. At the same time, the authors are aware of a recent proposal on bilevel optimization based coreset construction for continual learning [2]. [1] Q. Pham, D. Sahoo, C. Liu, and S. C. Hoi. Bilevel continual learning. arXiv preprint arXiv:2007.15553, 2020. [2] Z. Borsos, M. Mutny, and A. Krause. Coresets via bilevel optimization for continual learning and streaming. arXiv preprint arXiv:2006.03875, 2020

Related papers

• A Selective Learning Method for Temporal Graph Continual Learning [18.793135016181804]
  Real-life temporal graphs often introduce new node classes over time, but existing TGL methods assume a fixed set of classes. We define this novel problem as temporal graph continual learning (TGCL), which focuses on efficiently maintaining up-to-date knowledge of old classes. We derive an upper bound on the error caused by such replacement and transform it into objectives for selecting and learning subsets that minimize classification error while preserving the distribution of the full old-class data.
  arXiv  Detail & Related papers  (2025-03-03T14:22:20Z)
• Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Later [76.66498833720411]
  We introduce a differentiable version of $K$-nearest neighbors (KNN) originally designed to learn a linear projection to capture semantic similarities between instances. Surprisingly, our implementation of NCA using SGD and without dimensionality reduction already achieves decent performance on tabular data. We conclude our paper by analyzing the factors behind these improvements, including loss functions, prediction strategies, and deep architectures.
  arXiv  Detail & Related papers  (2024-07-03T16:38:57Z)
• A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation [121.0693322732454]
  Contrastive Language-Image Pretraining (CLIP) has gained popularity for its remarkable zero-shot capacity. Recent research has focused on developing efficient fine-tuning methods to enhance CLIP's performance in downstream tasks. We revisit a classical algorithm, Gaussian Discriminant Analysis (GDA), and apply it to the downstream classification of CLIP.
  arXiv  Detail & Related papers  (2024-02-06T15:45:27Z)
• Benchmarking Sensitivity of Continual Graph Learning for Skeleton-Based Action Recognition [6.14431765787048]
  Continual learning (CL) aims to build machine learning models that can accumulate knowledge continuously over different tasks without retraining from scratch. Previous studies have shown that pre-training graph neural networks (GNN) may lead to negative transfer after fine-tuning. We propose the first continual graph learning benchmark for continual graph learning setting.
  arXiv  Detail & Related papers  (2024-01-31T18:20:42Z)
• Automating Continual Learning [42.710124929514066]
  General-purpose learning systems should improve themselves in open-ended fashion in ever-changing environments. We propose Automated Continual Learning (ACL) to train self-referential neural networks to meta-learn their own in-context continual (meta-)learning algorithms.
  arXiv  Detail & Related papers  (2023-12-01T01:25:04Z)
• Accelerating Batch Active Learning Using Continual Learning Techniques [5.514154947870141]
  A major problem with Active Learning (AL) is high training costs since models are typically retrained from scratch after every query round. We develop a new class of techniques, circumventing this problem, by biasing further training towards previously labeled sets. We conduct experiments across many data domains, including natural language, vision, medical imaging, and computational biology.
  arXiv  Detail & Related papers  (2023-05-10T18:34:05Z)
• vCLIMB: A Novel Video Class Incremental Learning Benchmark [53.90485760679411]
  We introduce vCLIMB, a novel video continual learning benchmark. vCLIMB is a standardized test-bed to analyze catastrophic forgetting of deep models in video continual learning. We propose a temporal consistency regularization that can be applied on top of memory-based continual learning methods.
  arXiv  Detail & Related papers  (2022-01-23T22:14:17Z)
• Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data [101.6195176510611]
  "Online" continual learning enables evaluating both information retention and online learning efficacy. In online continual learning, each incoming small batch of data is first used for testing and then added to the training set, making the problem truly online. We introduce a new benchmark for online continual visual learning that exhibits large scale and natural distribution shifts.
  arXiv  Detail & Related papers  (2021-08-20T06:17:20Z)
• Does Continual Learning = Catastrophic Forgetting? [21.77693101142049]
  We present a set of tasks that surprisingly do not suffer from catastrophic forgetting when learned continually. We also introduce a novel yet simple algorithm, YASS, that outperforms state-of-the-art methods in the class-incremental categorization learning task. Finally, we present DyRT, a novel tool for tracking the dynamics of representation learning in continual models.
  arXiv  Detail & Related papers  (2021-01-18T19:29:12Z)
• Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical Guarantee [110.16183719936629]
  We propose an improved bilevel model which converges faster and better compared to the current formulation. The empirical results show that our model outperforms the current bilevel model with a great margin.
  arXiv  Detail & Related papers  (2020-09-01T20:52:57Z)
• Bilevel Continual Learning [76.50127663309604]
  We present a novel framework of continual learning named "Bilevel Continual Learning" (BCL) Our experiments on continual learning benchmarks demonstrate the efficacy of the proposed BCL compared to many state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-30T16:00:23Z)
• iTAML: An Incremental Task-Agnostic Meta-learning Approach [123.10294801296926]
  Humans can continuously learn new knowledge as their experience grows. Previous learning in deep neural networks can quickly fade out when they are trained on a new task. We introduce a novel meta-learning approach that seeks to maintain an equilibrium between all encountered tasks.
  arXiv  Detail & Related papers  (2020-03-25T21:42:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.