Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusion

• URL: http://arxiv.org/abs/2208.07978v2
• Date: Sat, 30 Sep 2023 04:32:58 GMT
• Title: Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusion
• Authors: Duy Phuong Nguyen, Sixing Yu, J. Pablo Mu\~noz, Ali Jannesari
• Abstract summary: This paper presents a new federated learning (FL) method that trains machine learning models on edge devices without accessing sensitive data. We propose a resource-aware FL method that aggregates local knowledge from edge models and distills it into robust global knowledge through knowledge distillation. Our method improves communication cost and performance in heterogeneous data and models compared to existing FL algorithms.
• Score: 9.106417025722756
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Concerned with user data privacy, this paper presents a new federated learning (FL) method that trains machine learning models on edge devices without accessing sensitive data. Traditional FL methods, although privacy-protective, fail to manage model heterogeneity and incur high communication costs due to their reliance on aggregation methods. To address this limitation, we propose a resource-aware FL method that aggregates local knowledge from edge models and distills it into robust global knowledge through knowledge distillation. This method allows efficient multi-model knowledge fusion and the deployment of resource-aware models while preserving model heterogeneity. Our method improves communication cost and performance in heterogeneous data and models compared to existing FL algorithms. Notably, it reduces the communication cost of ResNet-32 by up to 50\% and VGG-11 by up to 10$\times$ while delivering superior performance.

Related papers

• Towards Resource-Efficient Federated Learning in Industrial IoT for Multivariate Time Series Analysis [50.18156030818883]
  Anomaly and missing data constitute a thorny problem in industrial applications. Deep learning enabled anomaly detection has emerged as a critical direction. The data collected in edge devices contain user privacy.
  arXiv  Detail & Related papers  (2024-11-06T15:38:31Z)
• Data-Free Federated Class Incremental Learning with Diffusion-Based Generative Memory [27.651921957220004]
  We introduce a novel data-free federated class incremental learning framework with diffusion-based generative memory (DFedDGM) We design a new balanced sampler to help train the diffusion models to alleviate the common non-IID problem in FL. We also introduce an entropy-based sample filtering technique from an information theory perspective to enhance the quality of generative samples.
  arXiv  Detail & Related papers  (2024-05-22T20:59:18Z)
• Adaptive Hybrid Model Pruning in Federated Learning through Loss Exploration [17.589308358508863]
  We introduce AutoFLIP, an innovative approach that utilizes a federated loss exploration phase to drive adaptive hybrid pruning. We show that AutoFLIP not only efficiently accelerates global convergence, but also achieves superior accuracy and robustness compared to traditional methods.
  arXiv  Detail & Related papers  (2024-05-16T17:27:41Z)
• Fake It Till Make It: Federated Learning with Consensus-Oriented Generation [52.82176415223988]
  We propose federated learning with consensus-oriented generation (FedCOG) FedCOG consists of two key components at the client side: complementary data generation and knowledge-distillation-based model training. Experiments on classical and real-world FL datasets show that FedCOG consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-10T18:49:59Z)
• pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharing [19.403843478569303]
  We propose a model-heterogeneous personalized Federated learning approach based on feature extractor sharing. It incorporates a small homogeneous feature extractor into each client's heterogeneous local model. It achieves 1.61% higher test accuracy, while reducing communication and computation costs by 99.6% and 82.9%, respectively.
  arXiv  Detail & Related papers  (2023-11-12T15:43:39Z)
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning [86.59588262014456]
  Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG) Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
  arXiv  Detail & Related papers  (2022-03-17T11:18:17Z)
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning [61.488646649045215]
  Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
  arXiv  Detail & Related papers  (2021-11-28T19:03:39Z)
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation [56.886414139084216]
  Federated learning is widely used to learn intelligent models from decentralized data. In federated learning, clients need to communicate their local model updates in each iteration of model learning. We propose a communication efficient federated learning method based on knowledge distillation.
  arXiv  Detail & Related papers  (2021-08-30T15:39:54Z)
• Federated Robustness Propagation: Sharing Adversarial Robustness in Federated Learning [98.05061014090913]
  Federated learning (FL) emerges as a popular distributed learning schema that learns from a set of participating users without requiring raw data to be shared. adversarial training (AT) provides a sound solution for centralized learning, extending its usage for FL users has imposed significant challenges. We show that existing FL techniques cannot effectively propagate adversarial robustness among non-iid users. We propose a simple yet effective propagation approach that transfers robustness through carefully designed batch-normalization statistics.
  arXiv  Detail & Related papers  (2021-06-18T15:52:33Z)
• Data-Free Knowledge Distillation for Heterogeneous Federated Learning [31.364314540525218]
  Federated Learning (FL) is a decentralized machine-learning paradigm, in which a global server iteratively averages the model parameters of local users without accessing their data. Knowledge Distillation has recently emerged to tackle this issue, by refining the server model using aggregated knowledge from heterogeneous users. We propose a data-free knowledge distillation approach to address heterogeneous FL, where the server learns a lightweight generator to ensemble user information in a data-free manner.
  arXiv  Detail & Related papers  (2021-05-20T22:30:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.