Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generator

• URL: http://arxiv.org/abs/2407.11078v1
• Date: Sat, 13 Jul 2024 08:23:21 GMT
• Title: Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generator
• Authors: Thinh Nguyen, Khoa D Doan, Binh T. Nguyen, Danh Le-Phuoc, Kok-Seng Wong,
• Abstract summary: Federated Global Twin Generator (FedGTG) is an FCIL framework that exploits privacy-preserving generative-model training on the global side without accessing client data. We analyze the robustness of FedGTG on natural images, as well as its ability to converge to flat local minima and achieve better-predicting confidence (calibration) Experimental results on CIFAR-10, CIFAR-100, and tiny-ImageNet demonstrate the improvements in accuracy and forgetting measures of FedGTG compared to previous frameworks.
• Score: 13.808765929040677
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated Class-Incremental Learning (FCIL) increasingly becomes important in the decentralized setting, where it enables multiple participants to collaboratively train a global model to perform well on a sequence of tasks without sharing their private data. In FCIL, conventional Federated Learning algorithms such as FedAVG often suffer from catastrophic forgetting, resulting in significant performance declines on earlier tasks. Recent works, based on generative models, produce synthetic images to help mitigate this issue across all classes, but these approaches' testing accuracy on previous classes is still much lower than recent classes, i.e., having better plasticity than stability. To overcome these issues, this paper presents Federated Global Twin Generator (FedGTG), an FCIL framework that exploits privacy-preserving generative-model training on the global side without accessing client data. Specifically, the server trains a data generator and a feature generator to create two types of information from all seen classes, and then it sends the synthetic data to the client side. The clients then use feature-direction-controlling losses to make the local models retain knowledge and learn new tasks well. We extensively analyze the robustness of FedGTG on natural images, as well as its ability to converge to flat local minima and achieve better-predicting confidence (calibration). Experimental results on CIFAR-10, CIFAR-100, and tiny-ImageNet demonstrate the improvements in accuracy and forgetting measures of FedGTG compared to previous frameworks.

Related papers

• Recovering Global Data Distribution Locally in Federated Learning [7.885010255812708]
  Federated Learning (FL) is a distributed machine learning paradigm that enables collaboration among multiple clients. A major challenge in FL is the label imbalance, where clients may exclusively possess certain classes while having numerous minority and missing classes. We propose a novel approach ReGL to address this challenge, whose key idea is to Recover the Global data distribution Locally.
  arXiv  Detail & Related papers  (2024-09-21T08:35:04Z)
• Federated Class-Incremental Learning with Hierarchical Generative Prototypes [10.532838477096055]
  Federated Learning (FL) aims at unburdening the training of deep models by distributing computation across multiple devices (clients) Our proposal constrains both biases in the last layer by efficiently finetuning a pre-trained backbone using learnable prompts. Our method significantly improves the current State Of The Art, providing an average increase of +7.8% in accuracy.
  arXiv  Detail & Related papers  (2024-06-04T16:12:27Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• Client-side Gradient Inversion Against Federated Learning from Poisoning [59.74484221875662]
  Federated Learning (FL) enables distributed participants to train a global model without sharing data directly to a central server. Recent studies have revealed that FL is vulnerable to gradient inversion attack (GIA), which aims to reconstruct the original training samples. We propose Client-side poisoning Gradient Inversion (CGI), which is a novel attack method that can be launched from clients.
  arXiv  Detail & Related papers  (2023-09-14T03:48:27Z)
• Rethinking Client Drift in Federated Learning: A Logit Perspective [125.35844582366441]
  Federated Learning (FL) enables multiple clients to collaboratively learn in a distributed way, allowing for privacy protection. We find that the difference in logits between the local and global models increases as the model is continuously updated. We propose a new algorithm, named FedCSD, a Class prototype Similarity Distillation in a federated framework to align the local and global models.
  arXiv  Detail & Related papers  (2023-08-20T04:41:01Z)
• DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics [60.60173139258481]
  Local training on non-iid distributed data results in deflected local optimum. A natural solution is to gather all client data onto the server, such that the server has a global view of the entire data distribution. In this paper, we put forth an idea to collect and leverage global knowledge on the server without hindering data privacy.
  arXiv  Detail & Related papers  (2022-11-20T06:13:06Z)
• Federated Adversarial Training with Transformers [16.149924042225106]
  Federated learning (FL) has emerged to enable global model training over distributed clients' data while preserving its privacy. This paper investigates feasibility with different federated model aggregation methods and different vision transformer models with different tokenization and classification head techniques.
  arXiv  Detail & Related papers  (2022-06-05T09:07:09Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
  Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
  arXiv  Detail & Related papers  (2021-04-27T18:23:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.