The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation

• URL: http://arxiv.org/abs/2301.08968v2
• Date: Sun, 9 Apr 2023 15:03:20 GMT
• Title: The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation
• Authors: Huancheng Chen, Johnny (Chaining) Wang, Haris Vikalo
• Abstract summary: FedHKD (Federated Hyper-Knowledge Distillation) is a novel FL algorithm in which clients rely on knowledge distillation to train local models. Unlike other KD-based pFL methods, FedHKD does not rely on a public dataset nor it deploys a generative model at the server. We conduct extensive experiments on visual datasets in a variety of scenarios, demonstrating that FedHKD provides significant improvement in both personalized as well as global model performance.
• Score: 17.570719572024608
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Heterogeneity of data distributed across clients limits the performance of global models trained through federated learning, especially in the settings with highly imbalanced class distributions of local datasets. In recent years, personalized federated learning (pFL) has emerged as a potential solution to the challenges presented by heterogeneous data. However, existing pFL methods typically enhance performance of local models at the expense of the global model's accuracy. We propose FedHKD (Federated Hyper-Knowledge Distillation), a novel FL algorithm in which clients rely on knowledge distillation (KD) to train local models. In particular, each client extracts and sends to the server the means of local data representations and the corresponding soft predictions -- information that we refer to as ``hyper-knowledge". The server aggregates this information and broadcasts it to the clients in support of local training. Notably, unlike other KD-based pFL methods, FedHKD does not rely on a public dataset nor it deploys a generative model at the server. We analyze convergence of FedHKD and conduct extensive experiments on visual datasets in a variety of scenarios, demonstrating that FedHKD provides significant improvement in both personalized as well as global model performance compared to state-of-the-art FL methods designed for heterogeneous data settings.

Related papers

• Adversarial Federated Consensus Learning for Surface Defect Classification Under Data Heterogeneity in IIoT [8.48069043458347]
  It's difficult to collect and centralize sufficient training data from various entities in Industrial Internet of Things (IIoT) Federated learning (FL) provides a solution by enabling collaborative global model training across clients. We propose a novel personalized FL approach, named Adversarial Federated Consensus Learning (AFedCL)
  arXiv  Detail & Related papers  (2024-09-24T03:59:32Z)
• FedMAP: Unlocking Potential in Personalized Federated Learning through Bi-Level MAP Optimization [11.040916982022978]
  Federated Learning (FL) enables collaborative training of machine learning models on decentralized data. Data across clients often differs significantly due to class imbalance, feature distribution skew, sample size imbalance, and other phenomena. We propose a novel Bayesian PFL framework using bi-level optimization to tackle the data heterogeneity challenges.
  arXiv  Detail & Related papers  (2024-05-29T11:28:06Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning [20.135235291912185]
  Federated Learning (FL) is a privacy-constrained decentralized machine learning paradigm. We propose a new FL method (namely DFRD) to learn a robust global model in the data-heterogeneous and model-heterogeneous FL scenarios.
  arXiv  Detail & Related papers  (2023-09-24T04:29:22Z)
• Rethinking Client Drift in Federated Learning: A Logit Perspective [125.35844582366441]
  Federated Learning (FL) enables multiple clients to collaboratively learn in a distributed way, allowing for privacy protection. We find that the difference in logits between the local and global models increases as the model is continuously updated. We propose a new algorithm, named FedCSD, a Class prototype Similarity Distillation in a federated framework to align the local and global models.
  arXiv  Detail & Related papers  (2023-08-20T04:41:01Z)
• Visual Prompt Based Personalized Federated Learning [83.04104655903846]
  We propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients. Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms several state-of-the-art (SOTA) PFL algorithms by a large margin in various settings.
  arXiv  Detail & Related papers  (2023-03-15T15:02:15Z)
• Personalized Federated Learning with Hidden Information on Personalized Prior [18.8426865970643]
  We propose pFedBreD, a framework to solve the problem we model using Bregman divergence regularization. Our experiments show that our proposal significantly outcompetes other PFL algorithms on multiple public benchmarks.
  arXiv  Detail & Related papers  (2022-11-19T12:45:19Z)
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning [86.59588262014456]
  Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG) Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
  arXiv  Detail & Related papers  (2022-03-17T11:18:17Z)
• Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
  Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
  arXiv  Detail & Related papers  (2021-04-27T18:23:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.