Memory-efficient Continual Learning with Neural Collapse Contrastive

• URL: http://arxiv.org/abs/2412.02865v3
• Date: Fri, 06 Dec 2024 10:38:02 GMT
• Title: Memory-efficient Continual Learning with Neural Collapse Contrastive
• Authors: Trung-Anh Dang, Vincent Nguyen, Ngoc-Son Vu, Christel Vrain,
• Abstract summary: Contrastive learning has significantly improved representation quality, enhancing knowledge transfer across tasks in continual learning (CL) However, catastrophic forgetting remains a key challenge, as contrastive based methods primarily focus on "soft relationships" or "softness" between samples. We propose Focal Neural Collapse Contrastive (FNC2), a novel representation learning loss that effectively balances both soft and hard relationships.
• Score: 5.843533603338313
• License:
• Abstract: Contrastive learning has significantly improved representation quality, enhancing knowledge transfer across tasks in continual learning (CL). However, catastrophic forgetting remains a key challenge, as contrastive based methods primarily focus on "soft relationships" or "softness" between samples, which shift with changing data distributions and lead to representation overlap across tasks. Recently, the newly identified Neural Collapse phenomenon has shown promise in CL by focusing on "hard relationships" or "hardness" between samples and fixed prototypes. However, this approach overlooks "softness", crucial for capturing intra-class variability, and this rigid focus can also pull old class representations toward current ones, increasing forgetting. Building on these insights, we propose Focal Neural Collapse Contrastive (FNC^2), a novel representation learning loss that effectively balances both soft and hard relationships. Additionally, we introduce the Hardness-Softness Distillation (HSD) loss to progressively preserve the knowledge gained from these relationships across tasks. Our method outperforms state-of-the-art approaches, particularly in minimizing memory reliance. Remarkably, even without the use of memory, our approach rivals rehearsal-based methods, offering a compelling solution for data privacy concerns.

Related papers

• Exploring Contrastive Learning for Long-Tailed Multi-Label Text Classification [48.81069245141415]
  We introduce a novel contrastive loss function for multi-label text classification. It attains Micro-F1 scores that either match or surpass those obtained with other frequently employed loss functions. It demonstrates a significant improvement in Macro-F1 scores across three multi-label datasets.
  arXiv  Detail & Related papers  (2024-04-12T11:12:16Z)
• Relaxed Contrastive Learning for Federated Learning [48.96253206661268]
  We propose a novel contrastive learning framework to address the challenges of data heterogeneity in federated learning. Our framework outperforms all existing federated learning approaches by huge margins on the standard benchmarks.
  arXiv  Detail & Related papers  (2024-01-10T04:55:24Z)
• Primal Dual Continual Learning: Balancing Stability and Plasticity through Adaptive Memory Allocation [86.8475564814154]
  We show that it is both possible and beneficial to undertake the constrained optimization problem directly. We focus on memory-based methods, where a small subset of samples from previous tasks can be stored in a replay buffer. We show that dual variables indicate the sensitivity of the optimal value of the continual learning problem with respect to constraint perturbations.
  arXiv  Detail & Related papers  (2023-09-29T21:23:27Z)
• Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck [77.37409441129995]
  In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition. We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
  arXiv  Detail & Related papers  (2023-03-24T16:03:21Z)
• Mitigating Forgetting in Online Continual Learning via Contrasting Semantically Distinct Augmentations [22.289830907729705]
  Online continual learning (OCL) aims to enable model learning from a non-stationary data stream to continuously acquire new knowledge as well as retain the learnt one. Main challenge comes from the "catastrophic forgetting" issue -- the inability to well remember the learnt knowledge while learning the new ones.
  arXiv  Detail & Related papers  (2022-11-10T05:29:43Z)
• Consistent Representation Learning for Continual Relation Extraction [18.694012937149495]
  A consistent representation learning method is proposed, which maintains the stability of the relation embedding. Our method significantly outperforms state-of-the-art baselines and yield strong robustness on the imbalanced dataset.
  arXiv  Detail & Related papers  (2022-03-05T12:16:34Z)
• Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning [67.99349091593324]
  We investigate the relationship between the weight loss landscape and sensitivity-stability in the continual learning scenario. Our proposed method consistently outperforms baselines with the superior ability to learn new skills while alleviating forgetting effectively.
  arXiv  Detail & Related papers  (2021-10-09T15:13:44Z)
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation [68.47096022526927]
  Self-supervised learning has achieved great success in learning visual representations without data annotations. We introduce a novel relational SSL paradigm that learns representations by modeling the relationship between different instances. Our proposed ReSSL significantly outperforms the previous state-of-the-art algorithms in terms of both performance and training efficiency.
  arXiv  Detail & Related papers  (2021-07-20T06:53:07Z)
• Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping [10.970706194360451]
  Catastrophic forgetting in neural networks is a significant problem for continual learning. We introduce Relevance Mapping Networks (RMNs) which are inspired by the Optimal Overlap Hypothesis. We show that RMNs learn an optimized representational overlap that overcomes the twin problem of catastrophic forgetting and remembering.
  arXiv  Detail & Related papers  (2021-02-22T20:34:00Z)
• Enabling Continual Learning with Differentiable Hebbian Plasticity [18.12749708143404]
  Continual learning is the problem of sequentially learning new tasks or knowledge while protecting previously acquired knowledge. catastrophic forgetting poses a grand challenge for neural networks performing such learning process. We propose a Differentiable Hebbian Consolidation model which is composed of a Differentiable Hebbian Plasticity.
  arXiv  Detail & Related papers  (2020-06-30T06:42:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.