Flashbacks to Harmonize Stability and Plasticity in Continual Learning

• URL: http://arxiv.org/abs/2506.00477v1
• Date: Sat, 31 May 2025 09:04:58 GMT
• Title: Flashbacks to Harmonize Stability and Plasticity in Continual Learning
• Authors: Leila Mahmoodi, Peyman Moghadam, Munawar Hayat, Christian Simon, Mehrtash Harandi,
• Abstract summary: Flashback Learning (FL) is designed to harmonize the stability and plasticity of models in Continual Learning (CL)<n>FL operates through a two-phase training process and can be seamlessly integrated into various CL methods.
• Score: 36.686610963247794
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Flashback Learning (FL), a novel method designed to harmonize the stability and plasticity of models in Continual Learning (CL). Unlike prior approaches that primarily focus on regularizing model updates to preserve old information while learning new concepts, FL explicitly balances this trade-off through a bidirectional form of regularization. This approach effectively guides the model to swiftly incorporate new knowledge while actively retaining its old knowledge. FL operates through a two-phase training process and can be seamlessly integrated into various CL methods, including replay, parameter regularization, distillation, and dynamic architecture techniques. In designing FL, we use two distinct knowledge bases: one to enhance plasticity and another to improve stability. FL ensures a more balanced model by utilizing both knowledge bases to regularize model updates. Theoretically, we analyze how the FL mechanism enhances the stability-plasticity balance. Empirically, FL demonstrates tangible improvements over baseline methods within the same training budget. By integrating FL into at least one representative baseline from each CL category, we observed an average accuracy improvement of up to 4.91% in Class-Incremental and 3.51% in Task-Incremental settings on standard image classification benchmarks. Additionally, measurements of the stability-to-plasticity ratio confirm that FL effectively enhances this balance. FL also outperforms state-of-the-art CL methods on more challenging datasets like ImageNet.

Related papers

• Feasible Learning [78.6167929413604]
  We introduce Feasible Learning (FL), a sample-centric learning paradigm where models are trained by solving a feasibility problem that bounds the loss for each training sample.<n>Our empirical analysis, spanning image classification, age regression, and preference optimization in large language models, demonstrates that models trained via FL can learn from data while displaying improved tail behavior compared to ERM, with only a marginal impact on average performance.
  arXiv  Detail & Related papers  (2025-01-24T20:39:38Z)
• Unlocking the Potential of Model Calibration in Federated Learning [15.93119575317457]
  We propose Non-Uniform for Federated Learning (NUCFL), a generic framework that integrates FL with the concept of model calibration.<n>OurFL addresses this challenge by dynamically adjusting the model calibration based on relationships between each clients local model and the global model in FL.<n>By doing so,FL effectively aligns calibration needs for the global model while not sacrificing accuracy.
  arXiv  Detail & Related papers  (2024-09-07T20:11:11Z)
• Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning? [50.03434441234569]
  Federated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing. While various algorithms have shown that FL with local updates is a communication-efficient distributed learning framework, the generalization performance of FL with local updates has received comparatively less attention.
  arXiv  Detail & Related papers  (2024-09-05T19:00:18Z)
• On ADMM in Heterogeneous Federated Learning: Personalization, Robustness, and Fairness [16.595935469099306]
  We propose FLAME, an optimization framework by utilizing the alternating direction method of multipliers (ADMM) to train personalized and global models. Our theoretical analysis establishes the global convergence and two kinds of convergence rates for FLAME under mild assumptions. Our experimental findings show that FLAME outperforms state-of-the-art methods in convergence and accuracy, and it achieves higher test accuracy under various attacks.
  arXiv  Detail & Related papers  (2024-07-23T11:35:42Z)
• A Survey on Efficient Federated Learning Methods for Foundation Model Training [62.473245910234304]
  Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications.
  arXiv  Detail & Related papers  (2024-01-09T10:22:23Z)
• AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devices [61.66943750584406]
  We propose an Asynchronous Efficient Decentralized FL framework, i.e., AEDFL, in heterogeneous environments. First, we propose an asynchronous FL system model with an efficient model aggregation method for improving the FL convergence. Second, we propose a dynamic staleness-aware model update approach to achieve superior accuracy. Third, we propose an adaptive sparse training method to reduce communication and computation costs without significant accuracy degradation.
  arXiv  Detail & Related papers  (2023-12-18T05:18:17Z)
• Automated Federated Learning in Mobile Edge Networks -- Fast Adaptation and Convergence [83.58839320635956]
  Federated Learning (FL) can be used in mobile edge networks to train machine learning models in a distributed manner. Recent FL has been interpreted within a Model-Agnostic Meta-Learning (MAML) framework, which brings FL significant advantages in fast adaptation and convergence over heterogeneous datasets. This paper addresses how much benefit MAML brings to FL and how to maximize such benefit over mobile edge networks.
  arXiv  Detail & Related papers  (2023-03-23T02:42:10Z)
• Reliable Federated Disentangling Network for Non-IID Domain Feature [62.73267904147804]
  In this paper, we propose a novel reliable federated disentangling network, termed RFedDis. To the best of our knowledge, our proposed RFedDis is the first work to develop an FL approach based on evidential uncertainty combined with feature disentangling. Our proposed RFedDis provides outstanding performance with a high degree of reliability as compared to other state-of-the-art FL approaches.
  arXiv  Detail & Related papers  (2023-01-30T11:46:34Z)
• CyclicFL: A Cyclic Model Pre-Training Approach to Efficient Federated Learning [33.250038477336425]
  Federated learning (FL) has been proposed to enable distributed learning on Artificial Intelligence Internet of Things (AIoT) devices with guarantees of high-level data privacy. Existing FL methods suffer from both slow convergence and poor accuracy, especially in non-IID scenarios. We propose a novel method named CyclicFL, which can quickly derive effective initial models to guide the SGD processes.
  arXiv  Detail & Related papers  (2023-01-28T13:28:34Z)
• FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers [16.465900409973656]
  Federated Learning (FL) is an emerging paradigm that enables distributed users to collaboratively and iteratively train machine learning models without sharing their private data. Researchers are turning to using pre-trained Transformers instead of traditional convolutional neural networks in FL to leverage their excellent transfer learning capabilities. We show that fine-tuned Transformers achieve extraordinary performance on FL, and that the lightweight fine-tuning method facilitates a fast convergence rate and low communication costs.
  arXiv  Detail & Related papers  (2022-11-15T10:16:13Z)
• Federated Unlearning [24.60965999954735]
  Federated learning (FL) has emerged as a promising distributed machine learning (ML) paradigm. Practical needs of the "right to be forgotten" and countering data poisoning attacks call for efficient techniques that can remove, or unlearn, specific training data from the trained FL model. We present FedEraser, the first federated unlearning methodology that can eliminate the influence of a federated client's data on the global FL model.
  arXiv  Detail & Related papers  (2020-12-27T08:54:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.