Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning

• URL: http://arxiv.org/abs/2409.01128v2
• Date: Wed, 4 Sep 2024 02:40:52 GMT
• Title: Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning
• Authors: Jinglin Liang, Jin Zhong, Hanlin Gu, Zhongqi Lu, Xingxing Tang, Gang Dai, Shuangping Huang, Lixin Fan, Qiang Yang,
• Abstract summary: Key challenge in Federated Class Continual Learning is catastrophic forgetting. We propose a novel method of data replay based on diffusion models. Our method significantly outperforms existing baselines.
• Score: 13.836798036474143
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Class Continual Learning (FCCL) merges the challenges of distributed client learning with the need for seamless adaptation to new classes without forgetting old ones. The key challenge in FCCL is catastrophic forgetting, an issue that has been explored to some extent in Continual Learning (CL). However, due to privacy preservation requirements, some conventional methods, such as experience replay, are not directly applicable to FCCL. Existing FCCL methods mitigate forgetting by generating historical data through federated training of GANs or data-free knowledge distillation. However, these approaches often suffer from unstable training of generators or low-quality generated data, limiting their guidance for the model. To address this challenge, we propose a novel method of data replay based on diffusion models. Instead of training a diffusion model, we employ a pre-trained conditional diffusion model to reverse-engineer each class, searching the corresponding input conditions for each class within the model's input space, significantly reducing computational resources and time consumption while ensuring effective generation. Furthermore, we enhance the classifier's domain generalization ability on generated and real data through contrastive learning, indirectly improving the representational capability of generated data for real data. Comprehensive experiments demonstrate that our method significantly outperforms existing baselines. Code is available at https://github.com/jinglin-liang/DDDR.

Related papers

• TS-ACL: A Time Series Analytic Continual Learning Framework for Privacy-Preserving and Class-Incremental Pattern Recognition [14.6394894445113]
  We propose a Time Series Analytic Continual Learning framework, called TS-ACL. Inspired by analytical learning, TS-ACL transforms neural network updates into gradient-free linear regression problems. Our framework is highly suitable for real-time applications and large-scale data processing.
  arXiv  Detail & Related papers  (2024-10-21T12:34:02Z)
• Continual Domain Incremental Learning for Privacy-aware Digital Pathology [3.6630930118966814]
  Continual learning (CL) techniques aim to reduce the forgetting of past data when learning new data with distributional shift conditions. We develop a Generative Latent Replay-based CL (GLRCL) approach to store past data and perform latent replay with new data.
  arXiv  Detail & Related papers  (2024-09-10T12:21:54Z)
• Diffusion Model Meets Non-Exemplar Class-Incremental Learning and Beyond [48.51784137032964]
  Non-exemplar class-incremental learning (NECIL) is to resist catastrophic forgetting without saving old class samples. We propose a simple, yet effective textbfDiffusion-based textbfFeature textbfReplay (textbfDiffFR) method for NECIL.
  arXiv  Detail & Related papers  (2024-08-06T06:33:24Z)
• Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learning [22.13331870720021]
  We propose a beyond prompt learning approach to the RFCL task, called Continual Adapter (C-ADA) C-ADA flexibly extends specific weights in CAL to learn new knowledge for each task and freezes old weights to preserve prior knowledge. Our approach achieves significantly improved performance and training speed, outperforming the current state-of-the-art (SOTA) method.
  arXiv  Detail & Related papers  (2024-07-14T17:40:40Z)
• F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learning [28.772554281694166]
  Online Class Incremental Learning (OCIL) aims to train models incrementally, where data arrive in mini-batches, and previous data are not accessible. A major challenge in OCIL is Catastrophic Forgetting, i.e., the loss of previously learned knowledge. We propose an exemplar-free approach--Forward-only Online Analytic Learning (F-OAL) Unlike traditional methods, F-OAL does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time.
  arXiv  Detail & Related papers  (2024-03-23T07:39:13Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [100.7407460674153]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks. To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks. However, it is not expected in practice considering the memory constraint or data privacy issue. As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• Bypassing Logits Bias in Online Class-Incremental Learning with a Generative Framework [15.345043222622158]
  We focus on online class-incremental learning setting in which new classes emerge over time. Almost all existing methods are replay-based with a softmax classifier. We propose a novel generative framework based on the feature space.
  arXiv  Detail & Related papers  (2022-05-19T06:54:20Z)
• Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding [60.226644697970116]
  Domain classification is the fundamental task in natural language understanding (NLU) Most existing continual learning approaches suffer from low accuracy and performance fluctuation. We propose a hyper parameter-free continual learning model for text data that can stably produce high performance under various environments.
  arXiv  Detail & Related papers  (2022-01-05T02:46:16Z)
• Task-agnostic Continual Learning with Hybrid Probabilistic Models [75.01205414507243]
  We propose HCL, a Hybrid generative-discriminative approach to Continual Learning for classification. The flow is used to learn the data distribution, perform classification, identify task changes, and avoid forgetting. We demonstrate the strong performance of HCL on a range of continual learning benchmarks such as split-MNIST, split-CIFAR, and SVHN-MNIST.
  arXiv  Detail & Related papers  (2021-06-24T05:19:26Z)
• Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning [73.24988226158497]
  We consider the high-impact problem of Data-Free Class-Incremental Learning (DFCIL) We propose a novel incremental distillation strategy for DFCIL, contributing a modified cross-entropy training and importance-weighted feature distillation. Our method results in up to a 25.1% increase in final task accuracy (absolute difference) compared to SOTA DFCIL methods for common class-incremental benchmarks.
  arXiv  Detail & Related papers  (2021-06-17T17:56:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.