Continual Learning Using a Kernel-Based Method Over Foundation Models

• URL: http://arxiv.org/abs/2412.15571v1
• Date: Fri, 20 Dec 2024 05:09:18 GMT
• Title: Continual Learning Using a Kernel-Based Method Over Foundation Models
• Authors: Saleh Momeni, Sahisnu Mazumder, Bing Liu,
• Abstract summary: Class-incremental learning (CIL) learns a sequence of tasks incrementally.<n>CIL has two key challenges: catastrophic forgetting (CF) and inter-task class separation (ICS)<n>This paper proposes Kernel Linear Discriminant Analysis (KLDA) that can effectively avoid CF and ICS problems.
• Score: 13.315292874389735
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual learning (CL) learns a sequence of tasks incrementally. This paper studies the challenging CL setting of class-incremental learning (CIL). CIL has two key challenges: catastrophic forgetting (CF) and inter-task class separation (ICS). Despite numerous proposed methods, these issues remain persistent obstacles. This paper proposes a novel CIL method, called Kernel Linear Discriminant Analysis (KLDA), that can effectively avoid CF and ICS problems. It leverages only the powerful features learned in a foundation model (FM). However, directly using these features proves suboptimal. To address this, KLDA incorporates the Radial Basis Function (RBF) kernel and its Random Fourier Features (RFF) to enhance the feature representations from the FM, leading to improved performance. When a new task arrives, KLDA computes only the mean for each class in the task and updates a shared covariance matrix for all learned classes based on the kernelized features. Classification is performed using Linear Discriminant Analysis. Our empirical evaluation using text and image classification datasets demonstrates that KLDA significantly outperforms baselines. Remarkably, without relying on replay data, KLDA achieves accuracy comparable to joint training of all classes, which is considered the upper bound for CIL performance. The KLDA code is available at https://github.com/salehmomeni/klda.

Related papers

• KAC: Kolmogorov-Arnold Classifier for Continual Learning [70.29494592027852]
  Continual learning requires models to train continuously across consecutive tasks without forgetting. Most existing methods utilize linear classifiers, which struggle to maintain a stable classification space while learning new tasks. Inspired by the success of Kolmogorov-Arnold Networks (KAN) in preserving learning during simple continual regression tasks, we set out to explore their potential in more complex continual learning scenarios.
  arXiv  Detail & Related papers  (2025-03-27T01:27:14Z)
• Rethinking Few-shot 3D Point Cloud Semantic Segmentation [62.80639841429669]
  This paper revisits few-shot 3D point cloud semantic segmentation (FS-PCS) We focus on two significant issues in the state-of-the-art: foreground leakage and sparse point distribution. To address these issues, we introduce a standardized FS-PCS setting, upon which a new benchmark is built.
  arXiv  Detail & Related papers  (2024-03-01T15:14:47Z)
• A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation [121.0693322732454]
  Contrastive Language-Image Pretraining (CLIP) has gained popularity for its remarkable zero-shot capacity. Recent research has focused on developing efficient fine-tuning methods to enhance CLIP's performance in downstream tasks. We revisit a classical algorithm, Gaussian Discriminant Analysis (GDA), and apply it to the downstream classification of CLIP.
  arXiv  Detail & Related papers  (2024-02-06T15:45:27Z)
• FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning [21.088762527081883]
  Exemplar-free class-incremental learning (CIL) poses several challenges since it prohibits the rehearsal of data from previous tasks. Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention. We explore prototypical networks for CIL, which generate new class prototypes using the frozen feature extractor and classify the features based on the Euclidean distance to the prototypes.
  arXiv  Detail & Related papers  (2023-09-25T11:54:33Z)
• TKIL: Tangent Kernel Approach for Class Balanced Incremental Learning [4.822598110892847]
  Class incremental learning methods aim to keep a memory of a few exemplars from previously learned tasks, and distilling knowledge from them. Existing methods struggle to balance the performance across classes since they typically overfit the model to the latest task. We introduce a novel methodology of Tangent Kernel for Incremental Learning (TKIL) achieves that class-balanced performance.
  arXiv  Detail & Related papers  (2022-06-17T00:20:54Z)
• vCLIMB: A Novel Video Class Incremental Learning Benchmark [53.90485760679411]
  We introduce vCLIMB, a novel video continual learning benchmark. vCLIMB is a standardized test-bed to analyze catastrophic forgetting of deep models in video continual learning. We propose a temporal consistency regularization that can be applied on top of memory-based continual learning methods.
  arXiv  Detail & Related papers  (2022-01-23T22:14:17Z)
• Kernel Continual Learning [117.79080100313722]
  kernel continual learning is a simple but effective variant of continual learning to tackle catastrophic forgetting. episodic memory unit stores a subset of samples for each task to learn task-specific classifiers based on kernel ridge regression. variational random features to learn a data-driven kernel for each task.
  arXiv  Detail & Related papers  (2021-07-12T22:09:30Z)
• Few-Shot Incremental Learning with Continually Evolved Classifiers [46.278573301326276]
  Few-shot class-incremental learning (FSCIL) aims to design machine learning algorithms that can continually learn new concepts from a few data points. The difficulty lies in that limited data from new classes not only lead to significant overfitting issues but also exacerbate the notorious catastrophic forgetting problems. We propose a Continually Evolved CIF ( CEC) that employs a graph model to propagate context information between classifiers for adaptation.
  arXiv  Detail & Related papers  (2021-04-07T10:54:51Z)
• Learning Kernel for Conditional Moment-Matching Discrepancy-based Image Classification [26.09932710494144]
  A new kernel learning method is proposed to improve the discrimination performance of Conditional Maximum Mean Discrepancy (CMMD) It can be operated with deep network features iteratively and thus denoted as KLN for abbreviation. In particular, the kernel-based similarities are iteratively learned on the deep network features, and the algorithm can be implemented in an end-to-end manner.
  arXiv  Detail & Related papers  (2020-08-24T02:35:50Z)
• $\ell_p$-Norm Multiple Kernel One-Class Fisher Null-Space [15.000818334408802]
  The paper addresses the multiple kernel learning (MKL) problem for one-class classification (OCC) We present a multiple kernel learning algorithm where a general $ell_p$-norm constraint ($pgeq1$) on kernel weights is considered. An extension of the proposed one-class MKL approach is also considered where several related one-class MKL tasks are learned jointly.
  arXiv  Detail & Related papers  (2020-08-19T19:25:55Z)
• Federated Doubly Stochastic Kernel Learning for Vertically Partitioned Data [93.76907759950608]
  We propose a doubly kernel learning algorithm for vertically partitioned data. We show that FDSKL is significantly faster than state-of-the-art federated learning methods when dealing with kernels.
  arXiv  Detail & Related papers  (2020-08-14T05:46:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.