FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks

• URL: http://arxiv.org/abs/2201.03169v4
• Date: Tue, 01 Oct 2024 04:34:34 GMT
• Title: FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks
• Authors: Lingzhi Gao, Zhenyuan Zhang, Chao Wu,
• Abstract summary: We introduce a distributed three-player Generative Adversarial Network (GAN) to implement data-free mutual distillation. Our experiments on benchmark datasets vision demonstrate that our method outperforms other federated distillation algorithms in terms of generalization.
• Score: 4.1693122614074785
• License:
• Abstract: While existing federated learning approaches primarily focus on aggregating local models to construct a global model, in realistic settings, some clients may be reluctant to share their private models due to the inclusion of privacy-sensitive information. Knowledge distillation, which can extract model knowledge without accessing model parameters, is well-suited for this federated scenario. However, most distillation methods in federated learning (federated distillation) require a proxy dataset, which is difficult to obtain in the real world. Therefore, in this paper, we introduce a distributed three-player Generative Adversarial Network (GAN) to implement data-free mutual distillation and propose an effective method called FedDTG. We confirmed that the fake samples generated by GAN can make federated distillation more efficient and robust. Additionally, the distillation process between clients can deliver good individual client performance while simultaneously acquiring global knowledge and protecting data privacy. Our extensive experiments on benchmark vision datasets demonstrate that our method outperforms other federated distillation algorithms in terms of generalization.

Related papers

• Secure Federated Data Distillation [2.5311562666866494]
  We propose a Secure Federated Data Distillation framework (SFDD) to decentralize the distillation process while preserving privacy. Unlike existing Federated Distillation techniques that focus on training global models with distilled knowledge, our approach aims to produce a distilled dataset without exposing local contributions.
  arXiv  Detail & Related papers  (2025-02-19T13:54:44Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• Fake It Till Make It: Federated Learning with Consensus-Oriented Generation [52.82176415223988]
  We propose federated learning with consensus-oriented generation (FedCOG) FedCOG consists of two key components at the client side: complementary data generation and knowledge-distillation-based model training. Experiments on classical and real-world FL datasets show that FedCOG consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-10T18:49:59Z)
• Federated Learning Empowered by Generative Content [55.576885852501775]
  Federated learning (FL) enables leveraging distributed private data for model training in a privacy-preserving way. We propose a novel FL framework termed FedGC, designed to mitigate data heterogeneity issues by diversifying private data with generative content. We conduct a systematic empirical study on FedGC, covering diverse baselines, datasets, scenarios, and modalities.
  arXiv  Detail & Related papers  (2023-12-10T07:38:56Z)
• Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents [43.282328554697564]
  We propose a highly efficient FL dataset distillation framework on the server side. Unlike previous strategies, our technique enables the server to leverage prior knowledge from pre-trained deep generative models. Our framework converges faster than the baselines because rather than the server trains on several sets of heterogeneous data distributions, it trains on a multi-modal distribution.
  arXiv  Detail & Related papers  (2023-12-03T23:30:48Z)
• Federated Learning on Non-iid Data via Local and Global Distillation [25.397058380098816]
  We propose FedND: federated learning with noise distillation. In the client, we propose a self-distillation method to train the local model. In the server, we generate noisy samples for each client and use them to distill other clients. Experimental results show that the algorithm achieves the best performance and is more communication-efficient than state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-06-26T06:14:01Z)
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning [86.59588262014456]
  Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG) Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
  arXiv  Detail & Related papers  (2022-03-17T11:18:17Z)
• Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
  Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
  arXiv  Detail & Related papers  (2021-04-27T18:23:54Z)
• Toward Understanding the Influence of Individual Clients in Federated Learning [52.07734799278535]
  Federated learning allows clients to jointly train a global model without sending their private data to a central server. We defined a new notion called em-Influence, quantify this influence over parameters, and proposed an effective efficient model to estimate this metric.
  arXiv  Detail & Related papers  (2020-12-20T14:34:36Z)
• Adaptive Distillation for Decentralized Learning from Heterogeneous Clients [9.261720698142097]
  We propose a new decentralized learning method called Decentralized Learning via Adaptive Distillation (DLAD) The proposed DLAD aggregates the outputs of the client models while adaptively emphasizing those with higher confidence in given distillation samples. Our extensive experimental evaluation on multiple public datasets demonstrates the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2020-08-18T14:25:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.