LSEBMCL: A Latent Space Energy-Based Model for Continual Learning

• URL: http://arxiv.org/abs/2501.05495v1
• Date: Thu, 09 Jan 2025 15:47:30 GMT
• Title: LSEBMCL: A Latent Space Energy-Based Model for Continual Learning
• Authors: Xiaodi Li, Dingcheng Li, Rujun Gao, Mahmoud Zamani, Latifur Khan,
• Abstract summary: The study demonstrates the efficacy of EBM in NLP tasks, achieving state-of-the-art results in all experiments. The proposed solution LSEBMCL (Latent Space Energy-Based Model for Continual Learning) in this work is to use energy-based models (EBMs) to prevent catastrophic forgetting.
• Score: 20.356926275395004
• License:
• Abstract: Continual learning has become essential in many practical applications such as online news summaries and product classification. The primary challenge is known as catastrophic forgetting, a phenomenon where a model inadvertently discards previously learned knowledge when it is trained on new tasks. Existing solutions involve storing exemplars from previous classes, regularizing parameters during the fine-tuning process, or assigning different model parameters to each task. The proposed solution LSEBMCL (Latent Space Energy-Based Model for Continual Learning) in this work is to use energy-based models (EBMs) to prevent catastrophic forgetting by sampling data points from previous tasks when training on new ones. The EBM is a machine learning model that associates an energy value with each input data point. The proposed method uses an EBM layer as an outer-generator in the continual learning framework for NLP tasks. The study demonstrates the efficacy of EBM in NLP tasks, achieving state-of-the-art results in all experiments.

Related papers

• Unlearning with Control: Assessing Real-world Utility for Large Language Model Unlearning [97.2995389188179]
  Recent research has begun to approach large language models (LLMs) unlearning via gradient ascent (GA) Despite their simplicity and efficiency, we suggest that GA-based methods face the propensity towards excessive unlearning. We propose several controlling methods that can regulate the extent of excessive unlearning.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
  Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information. We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting. Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
  arXiv  Detail & Related papers  (2023-09-15T17:10:51Z)
• PILOT: A Pre-Trained Model-Based Continual Learning Toolbox [65.57123249246358]
  This paper introduces a pre-trained model-based continual learning toolbox known as PILOT. On the one hand, PILOT implements some state-of-the-art class-incremental learning algorithms based on pre-trained models, such as L2P, DualPrompt, and CODA-Prompt. On the other hand, PILOT fits typical class-incremental learning algorithms within the context of pre-trained models to evaluate their effectiveness.
  arXiv  Detail & Related papers  (2023-09-13T17:55:11Z)
• Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning [40.13416912075668]
  We propose a rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks. Our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order.
  arXiv  Detail & Related papers  (2023-06-21T01:43:25Z)
• SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities [76.97949110580703]
  We introduce SUPERB-SG, a new benchmark to evaluate pre-trained models across various speech tasks. We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation.
  arXiv  Detail & Related papers  (2022-03-14T04:26:40Z)
• Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning [54.7584721943286]
  Continual Learning (CL) methods aim to enable machine learning models to learn new tasks without catastrophic forgetting of those that have been previously mastered. Existing CL approaches often keep a buffer of previously-seen samples, perform knowledge distillation, or use regularization techniques towards this goal. We propose to only activate and select sparse neurons for learning current and past tasks at any stage.
  arXiv  Detail & Related papers  (2022-02-21T13:25:03Z)
• $\mathcal{F}$-EBM: Energy Based Learning of Functional Data [1.0896567381206714]
  Energy-Based Models (EBMs) have proven to be a highly effective approach for modelling densities on finite-dimensional spaces. We present a novel class of EBM which is able to learn distributions of functions from functional samples evaluated at finitely many points.
  arXiv  Detail & Related papers  (2022-02-04T01:01:50Z)
• Learning Proposals for Practical Energy-Based Regression [46.05502630457458]
  Energy-based models (EBMs) have experienced a resurgence within machine learning in recent years. We introduce a conceptually simple method to automatically learn an effective proposal distribution, which is parameterized by a separate network head. At test-time, we can then employ importance sampling with the trained proposal to efficiently evaluate the learned EBM and produce stand-alone predictions.
  arXiv  Detail & Related papers  (2021-10-22T17:58:05Z)
• Energy-Efficient and Federated Meta-Learning via Projected Stochastic Gradient Ascent [79.58680275615752]
  We propose an energy-efficient federated meta-learning framework. We assume each task is owned by a separate agent, so a limited number of tasks is used to train a meta-model.
  arXiv  Detail & Related papers  (2021-05-31T08:15:44Z)
• Energy-Based Models for Continual Learning [36.05297743063411]
  We motivate Energy-Based Models (EBMs) as a promising model class for continual learning problems. Our proposed version of EBMs for continual learning is simple, efficient and outperforms baseline methods by a large margin on several benchmarks.
  arXiv  Detail & Related papers  (2020-11-24T17:08:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.