DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning

• URL: http://arxiv.org/abs/2403.17503v1
• Date: Tue, 26 Mar 2024 09:04:18 GMT
• Title: DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning
• Abstract summary: Class-incremental learning (CIL) under an exemplar-free constraint has presented a significant challenge. We propose a Dual-Stream Analytic Learning (DS-AL) approach to solve the exemplar-free CIL problem. Empirical results demonstrate that the DS-AL, despite being an exemplar-free technique, delivers performance comparable with or better than that of replay-based methods.
• Score: 27.585993693155217
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Class-incremental learning (CIL) under an exemplar-free constraint has presented a significant challenge. Existing methods adhering to this constraint are prone to catastrophic forgetting, far more so than replay-based techniques that retain access to past samples. In this paper, to solve the exemplar-free CIL problem, we propose a Dual-Stream Analytic Learning (DS-AL) approach. The DS-AL contains a main stream offering an analytical (i.e., closed-form) linear solution, and a compensation stream improving the inherent under-fitting limitation due to adopting linear mapping. The main stream redefines the CIL problem into a Concatenated Recursive Least Squares (C-RLS) task, allowing an equivalence between the CIL and its joint-learning counterpart. The compensation stream is governed by a Dual-Activation Compensation (DAC) module. This module re-activates the embedding with a different activation function from the main stream one, and seeks fitting compensation by projecting the embedding to the null space of the main stream's linear mapping. Empirical results demonstrate that the DS-AL, despite being an exemplar-free technique, delivers performance comparable with or better than that of replay-based methods across various datasets, including CIFAR-100, ImageNet-100 and ImageNet-Full. Additionally, the C-RLS' equivalent property allows the DS-AL to execute CIL in a phase-invariant manner. This is evidenced by a never-before-seen 500-phase CIL ImageNet task, which performs on a level identical to a 5-phase one. Our codes are available at https://github.com/ZHUANGHP/Analytic-continual-learning.

Related papers

• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models [37.194177406897964]
  We propose a closed-form formulation that computes the linear probe induced by the synthetic set with a sample-space kernel ridge solver.<n>On ImageNet-100 with four pre-trained backbones, CLP-DD substantially improves over LGM without DSA and approaches LGM with DSA at a fraction of the computational cost.<n>On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA on three of four backbones while running roughly $14times$ faster and using less than one-eighth of the GPU memory.
  arXiv  Detail & Related papers  (2026-05-08T03:41:51Z)
• Towards Realistic Class-Incremental Learning with Free-Flow Increments [17.558920457942936]
  We propose a model-agnostic framework for robust CIL learning under free-flow arrivals.<n>It comprises a class-wise mean (CWM) objective that replaces sample frequency weighted loss with uniformly aggregated class-conditional supervision.<n>Experiments confirm a clear performance degradation across various CIL baselines under FFCIL.
  arXiv  Detail & Related papers  (2026-04-03T06:19:58Z)
• Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards [76.49428173793386]
  LLMs must invoke multiple dependent APIs in the correct order while propagating intermediate outputs.<n>Training models to handle such faces two obstacles: existing environments focus on simple per-turn function calls with simulated data, and binary rewards.<n>We present a framework addressing both challenges. First, we construct a reinforcement learning environment backed by a large-scale cache of real API responses, enabling a data synthesis pipeline that samples valid multi-step orchestration traces.<n>Second, we propose a graduated reward design that decomposes correctness into atomic validity and orchestration.
  arXiv  Detail & Related papers  (2026-03-25T18:31:39Z)
• ResAD++: Towards Class Agnostic Anomaly Detection via Residual Feature Learning [52.11294707895649]
  This paper explores the problem of class-agnostic anomaly detection (AD)<n>The objective is to train one class-agnostic AD model that can generalize to detect anomalies in diverse new classes from different domains without any retraining or fine-tuning on the target data.<n> Comprehensive experiments on eight real-world AD datasets demonstrate that our ResAD++ can achieve remarkable AD results when directly used in new classes.
  arXiv  Detail & Related papers  (2025-09-28T08:41:05Z)
• Online Analytic Exemplar-Free Continual Learning with Large Models for Imbalanced Autonomous Driving Task [25.38082751323396]
  We propose an Analytic Exemplar-Free Online Continual Learning algorithm (AEF-OCL) The AEF-OCL leverages analytic continual learning principles and employs ridge regression as a classifier for features extracted by a large backbone network. Experimental results demonstrate that despite being an exemplar-free strategy, our method outperforms various methods on the autonomous driving SODA10M dataset.
  arXiv  Detail & Related papers  (2024-05-28T03:19:15Z)
• REAL: Representation Enhanced Analytic Learning for Exemplar-free Class-incremental Learning [12.197327462627912]
  We propose a representation enhanced analytic learning (REAL) for Exemplar-free class-incremental learning (EFCIL) The REAL constructs a dual-stream base pretraining (DS-BPT) and a representation enhancing distillation (RED) process to enhance the representation of the extractor. Our method addresses the issue of insufficient discriminability in representations of unseen data caused by a frozen backbone in the existing AL-based CIL.
  arXiv  Detail & Related papers  (2024-03-20T11:48:10Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• Dynamic Residual Classifier for Class Incremental Learning [4.02487511510606]
  With imbalanced sample numbers between old and new classes, the learning can be biased. Existing CIL methods exploit the longtailed (LT) recognition techniques, e.g., the adjusted losses and the data re-sampling methods. A novel Dynamic Residual adaptation (DRC) is proposed to handle this challenging scenario.
  arXiv  Detail & Related papers  (2023-08-25T11:07:11Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• Dealing with Cross-Task Class Discrimination in Online Continual Learning [54.31411109376545]
  This paper argues for another challenge in class-incremental learning (CIL) How to establish decision boundaries between the classes of the new task and old tasks with no (or limited) access to the old task data. A replay method saves a small amount of data (replay data) from previous tasks. When a batch of current task data arrives, the system jointly trains the new data and some sampled replay data. This paper argues that the replay approach also has a dynamic training bias issue which reduces the effectiveness of the replay data in solving the CTCD problem.
  arXiv  Detail & Related papers  (2023-05-24T02:52:30Z)
• DELTA: Dynamic Embedding Learning with Truncated Conscious Attention for CTR Prediction [61.68415731896613]
  Click-Through Rate (CTR) prediction is a pivotal task in product and content recommendation. We propose a model that enables Dynamic Embedding Learning with Truncated Conscious Attention for CTR prediction.
  arXiv  Detail & Related papers  (2023-05-03T12:34:45Z)
• Slimmable Networks for Contrastive Self-supervised Learning [69.9454691873866]
  Self-supervised learning makes significant progress in pre-training large models, but struggles with small models. We introduce another one-stage solution to obtain pre-trained small models without the need for extra teachers. A slimmable network consists of a full network and several weight-sharing sub-networks, which can be pre-trained once to obtain various networks.
  arXiv  Detail & Related papers  (2022-09-30T15:15:05Z)
• Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods [19.587273175563745]
  Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations. This paper proposes a unifying framework under the helm of spectral manifold learning to address those limitations.
  arXiv  Detail & Related papers  (2022-05-23T17:59:32Z)
• Class Incremental Online Streaming Learning [40.97848249237289]
  We propose a novel approach for the class-incremental learning in an emphonline streaming setting to address these challenges. The proposed approach leverages implicit and explicit dual weight regularization and experience replay. Also, we propose an efficient online memory replay and replacement buffer strategy that significantly boosts the model's performance.
  arXiv  Detail & Related papers  (2021-10-20T19:24:31Z)
• Contrastive Prototype Learning with Augmented Embeddings for Few-Shot Learning [58.2091760793799]
  We propose a novel contrastive prototype learning with augmented embeddings (CPLAE) model. With a class prototype as an anchor, CPL aims to pull the query samples of the same class closer and those of different classes further away. Extensive experiments on several benchmarks demonstrate that our proposed CPLAE achieves new state-of-the-art.
  arXiv  Detail & Related papers  (2021-01-23T13:22:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.