DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning

• URL: http://arxiv.org/abs/2403.17503v1
• Date: Tue, 26 Mar 2024 09:04:18 GMT
• Title: DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning
• Authors: Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, Zhiping Lin,
• Abstract summary: Class-incremental learning (CIL) under an exemplar-free constraint has presented a significant challenge. We propose a Dual-Stream Analytic Learning (DS-AL) approach to solve the exemplar-free CIL problem. Empirical results demonstrate that the DS-AL, despite being an exemplar-free technique, delivers performance comparable with or better than that of replay-based methods.
• Score: 27.585993693155217
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Class-incremental learning (CIL) under an exemplar-free constraint has presented a significant challenge. Existing methods adhering to this constraint are prone to catastrophic forgetting, far more so than replay-based techniques that retain access to past samples. In this paper, to solve the exemplar-free CIL problem, we propose a Dual-Stream Analytic Learning (DS-AL) approach. The DS-AL contains a main stream offering an analytical (i.e., closed-form) linear solution, and a compensation stream improving the inherent under-fitting limitation due to adopting linear mapping. The main stream redefines the CIL problem into a Concatenated Recursive Least Squares (C-RLS) task, allowing an equivalence between the CIL and its joint-learning counterpart. The compensation stream is governed by a Dual-Activation Compensation (DAC) module. This module re-activates the embedding with a different activation function from the main stream one, and seeks fitting compensation by projecting the embedding to the null space of the main stream's linear mapping. Empirical results demonstrate that the DS-AL, despite being an exemplar-free technique, delivers performance comparable with or better than that of replay-based methods across various datasets, including CIFAR-100, ImageNet-100 and ImageNet-Full. Additionally, the C-RLS' equivalent property allows the DS-AL to execute CIL in a phase-invariant manner. This is evidenced by a never-before-seen 500-phase CIL ImageNet task, which performs on a level identical to a 5-phase one. Our codes are available at https://github.com/ZHUANGHP/Analytic-continual-learning.

Related papers

• Online Analytic Exemplar-Free Continual Learning with Large Models for Imbalanced Autonomous Driving Task [25.38082751323396]
  We propose an Analytic Exemplar-Free Online Continual Learning algorithm (AEF-OCL) The AEF-OCL leverages analytic continual learning principles and employs ridge regression as a classifier for features extracted by a large backbone network. Experimental results demonstrate that despite being an exemplar-free strategy, our method outperforms various methods on the autonomous driving SODA10M dataset.
  arXiv  Detail & Related papers  (2024-05-28T03:19:15Z)
• REAL: Representation Enhanced Analytic Learning for Exemplar-free Class-incremental Learning [12.197327462627912]
  We propose a representation enhanced analytic learning (REAL) for Exemplar-free class-incremental learning (EFCIL) The REAL constructs a dual-stream base pretraining (DS-BPT) and a representation enhancing distillation (RED) process to enhance the representation of the extractor. Our method addresses the issue of insufficient discriminability in representations of unseen data caused by a frozen backbone in the existing AL-based CIL.
  arXiv  Detail & Related papers  (2024-03-20T11:48:10Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• Dynamic Residual Classifier for Class Incremental Learning [4.02487511510606]
  With imbalanced sample numbers between old and new classes, the learning can be biased. Existing CIL methods exploit the longtailed (LT) recognition techniques, e.g., the adjusted losses and the data re-sampling methods. A novel Dynamic Residual adaptation (DRC) is proposed to handle this challenging scenario.
  arXiv  Detail & Related papers  (2023-08-25T11:07:11Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• Dealing with Cross-Task Class Discrimination in Online Continual Learning [54.31411109376545]
  This paper argues for another challenge in class-incremental learning (CIL) How to establish decision boundaries between the classes of the new task and old tasks with no (or limited) access to the old task data. A replay method saves a small amount of data (replay data) from previous tasks. When a batch of current task data arrives, the system jointly trains the new data and some sampled replay data. This paper argues that the replay approach also has a dynamic training bias issue which reduces the effectiveness of the replay data in solving the CTCD problem.
  arXiv  Detail & Related papers  (2023-05-24T02:52:30Z)
• DELTA: Dynamic Embedding Learning with Truncated Conscious Attention for CTR Prediction [61.68415731896613]
  Click-Through Rate (CTR) prediction is a pivotal task in product and content recommendation. We propose a model that enables Dynamic Embedding Learning with Truncated Conscious Attention for CTR prediction.
  arXiv  Detail & Related papers  (2023-05-03T12:34:45Z)
• Slimmable Networks for Contrastive Self-supervised Learning [69.9454691873866]
  Self-supervised learning makes significant progress in pre-training large models, but struggles with small models. We introduce another one-stage solution to obtain pre-trained small models without the need for extra teachers. A slimmable network consists of a full network and several weight-sharing sub-networks, which can be pre-trained once to obtain various networks.
  arXiv  Detail & Related papers  (2022-09-30T15:15:05Z)
• Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods [19.587273175563745]
  Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations. This paper proposes a unifying framework under the helm of spectral manifold learning to address those limitations.
  arXiv  Detail & Related papers  (2022-05-23T17:59:32Z)
• Class Incremental Online Streaming Learning [40.97848249237289]
  We propose a novel approach for the class-incremental learning in an emphonline streaming setting to address these challenges. The proposed approach leverages implicit and explicit dual weight regularization and experience replay. Also, we propose an efficient online memory replay and replacement buffer strategy that significantly boosts the model's performance.
  arXiv  Detail & Related papers  (2021-10-20T19:24:31Z)
• Contrastive Prototype Learning with Augmented Embeddings for Few-Shot Learning [58.2091760793799]
  We propose a novel contrastive prototype learning with augmented embeddings (CPLAE) model. With a class prototype as an anchor, CPL aims to pull the query samples of the same class closer and those of different classes further away. Extensive experiments on several benchmarks demonstrate that our proposed CPLAE achieves new state-of-the-art.
  arXiv  Detail & Related papers  (2021-01-23T13:22:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.