ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes

• URL: http://arxiv.org/abs/2412.11207v1
• Date: Sun, 15 Dec 2024 14:49:29 GMT
• Title: ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes
• Authors: Pedro Miguel Sánchez Sánchez, Enrique Tomás Martínez Beltrán, Miguel Fernández Llamas, Gérôme Bovet, Gregorio Martínez Pérez, Alberto Huertas Celdrán,
• Abstract summary: Decentralized Federated Learning (DFL) trains models in a collaborative and privacy-preserving manner. This paper introduces ProFe, a novel communication optimization algorithm for DFL that combines knowledge distillation, prototype learning, and quantization techniques.
• Score: 3.7340128675975173
• License:
• Abstract: Decentralized Federated Learning (DFL) trains models in a collaborative and privacy-preserving manner while removing model centralization risks and improving communication bottlenecks. However, DFL faces challenges in efficient communication management and model aggregation within decentralized environments, especially with heterogeneous data distributions. Thus, this paper introduces ProFe, a novel communication optimization algorithm for DFL that combines knowledge distillation, prototype learning, and quantization techniques. ProFe utilizes knowledge from large local models to train smaller ones for aggregation, incorporates prototypes to better learn unseen classes, and applies quantization to reduce data transmitted during communication rounds. The performance of ProFe has been validated and compared to the literature by using benchmark datasets like MNIST, CIFAR10, and CIFAR100. Results showed that the proposed algorithm reduces communication costs by up to ~40-50% while maintaining or improving model performance. In addition, it adds ~20% training time due to increased complexity, generating a trade-off.

Related papers

• Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models [43.26028399395612]
  We propose a Fisher Information-based Efficient Curriculum Federated Learning framework (FibecFed) with two novel methods. First, we propose a fisher information-based method to adaptively sample data within each device to improve the effectiveness of the FL fine-tuning process. Second, we dynamically select the proper layers for global aggregation and sparse parameters for local update with LoRA.
  arXiv  Detail & Related papers  (2024-09-30T18:12:18Z)
• FedLog: Personalized Federated Classification with Less Communication and More Flexibility [24.030147353437382]
  Federated representation learning (FRL) aims to learn personalized federated models with effective feature extraction from local data. To reduce the overhead, we propose to share sufficient data summaries instead of raw model parameters.
  arXiv  Detail & Related papers  (2024-07-11T09:40:29Z)
• Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networks [72.59891661768177]
  Federated learning (FL) enables distributed learning across edge devices while protecting data privacy. We consider a FL framework with partial model pruning and personalization to overcome these challenges. This framework splits the learning model into a global part with model pruning shared with all devices to learn data representations and a personalized part to be fine-tuned for a specific device.
  arXiv  Detail & Related papers  (2023-09-04T21:10:45Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)
• DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training [84.81043932706375]
  We propose a novel personalized federated learning framework in a decentralized (peer-to-peer) communication protocol named Dis-PFL. Dis-PFL employs personalized sparse masks to customize sparse local models on the edge. We demonstrate that our method can easily adapt to heterogeneous local clients with varying computation complexities.
  arXiv  Detail & Related papers  (2022-06-01T02:20:57Z)
• FedCAT: Towards Accurate Federated Learning via Device Concatenation [4.416919766772866]
  Federated Learning (FL) enables all the involved devices to train a global model collaboratively without exposing their local data privacy. For non-IID scenarios, the classification accuracy of FL models decreases drastically due to the weight divergence caused by data heterogeneity. We introduce a novel FL approach named Fed-Cat that can achieve high model accuracy based on our proposed device selection strategy and device concatenation-based local training method.
  arXiv  Detail & Related papers  (2022-02-23T10:08:43Z)
• Federated Learning over Wireless IoT Networks with Optimized Communication and Resources [98.18365881575805]
  Federated learning (FL) as a paradigm of collaborative learning techniques has obtained increasing research attention. It is of interest to investigate fast responding and accurate FL schemes over wireless systems. We show that the proposed communication-efficient federated learning framework converges at a strong linear rate.
  arXiv  Detail & Related papers  (2021-10-22T13:25:57Z)
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation [56.886414139084216]
  Federated learning is widely used to learn intelligent models from decentralized data. In federated learning, clients need to communicate their local model updates in each iteration of model learning. We propose a communication efficient federated learning method based on knowledge distillation.
  arXiv  Detail & Related papers  (2021-08-30T15:39:54Z)
• Communication-Efficient Hierarchical Federated Learning for IoT Heterogeneous Systems with Imbalanced Data [42.26599494940002]
  Federated learning (FL) is a distributed learning methodology that allows multiple nodes to cooperatively train a deep learning model. This paper studies the potential of hierarchical FL in IoT heterogeneous systems. It proposes an optimized solution for user assignment and resource allocation on multiple edge nodes.
  arXiv  Detail & Related papers  (2021-07-14T08:32:39Z)
• CosSGD: Nonlinear Quantization for Communication-efficient Federated Learning [62.65937719264881]
  Federated learning facilitates learning across clients without transferring local data on these clients to a central server. We propose a nonlinear quantization for compressed gradient descent, which can be easily utilized in federated learning. Our system significantly reduces the communication cost by up to three orders of magnitude, while maintaining convergence and accuracy of the training process.
  arXiv  Detail & Related papers  (2020-12-15T12:20:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.