FedCode: Communication-Efficient Federated Learning via Transferring Codebooks

• URL: http://arxiv.org/abs/2311.09270v1
• Date: Wed, 15 Nov 2023 12:06:32 GMT
• Title: FedCode: Communication-Efficient Federated Learning via Transferring Codebooks
• Authors: Saeed Khalilian, Vasileios Tsouvalas, Tanir Ozcelebi, Nirvana Meratnia
• Abstract summary: Federated Learning (FL) is a distributed machine learning paradigm that enables learning models from decentralized local data. Existing approaches rely on model compression techniques, such as pruning and weight clustering to tackle this. We propose FedCode where clients transmit only codebooks, i.e., the cluster centers of updated model weight values.
• Score: 3.004066195320147
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) is a distributed machine learning paradigm that enables learning models from decentralized local data. While FL offers appealing properties for clients' data privacy, it imposes high communication burdens for exchanging model weights between a server and the clients. Existing approaches rely on model compression techniques, such as pruning and weight clustering to tackle this. However, transmitting the entire set of weight updates at each federated round, even in a compressed format, limits the potential for a substantial reduction in communication volume. We propose FedCode where clients transmit only codebooks, i.e., the cluster centers of updated model weight values. To ensure a smooth learning curve and proper calibration of clusters between the server and the clients, FedCode periodically transfers model weights after multiple rounds of solely communicating codebooks. This results in a significant reduction in communication volume between clients and the server in both directions, without imposing significant computational overhead on the clients or leading to major performance degradation of the models. We evaluate the effectiveness of FedCode using various publicly available datasets with ResNet-20 and MobileNet backbone model architectures. Our evaluations demonstrate a 12.2-fold data transmission reduction on average while maintaining a comparable model performance with an average accuracy loss of 1.3% compared to FedAvg. Further validation of FedCode performance under non-IID data distributions showcased an average accuracy loss of 2.0% compared to FedAvg while achieving approximately a 12.7-fold data transmission reduction.

Related papers

• FedX: Explanation-Guided Pruning for Communication-Efficient Federated Learning in Remote Sensing [2.725507329935916]
  Federated learning is a suitable learning paradigm for remote sensing (RS) image classification tasks.<n>A key challenge in applying FL to RS tasks is the communication overhead caused by the frequent exchange of large model updates between clients and the central server.<n>We propose a novel strategy (denoted as FedX) that uses explanation-guided pruning to reduce communication overhead.
  arXiv  Detail & Related papers  (2025-08-08T12:21:58Z)
• FedBWO: Enhancing Communication Efficiency in Federated Learning [4.788163807490197]
  Federated Learning (FL) is a distributed Machine Learning (ML) setup, where a shared model is collaboratively trained by various clients using their local datasets while keeping the data private.<n>Considering resource constraints, increasing the number of clients and the amount of data (model weights) can lead to a bottleneck.<n>In this paper, we introduce the Federated Black Widow Optimization (FedBWO) technique to decrease the amount of transmitted data by transmitting only a performance score rather than the local model weights from clients.
  arXiv  Detail & Related papers  (2025-05-07T14:02:35Z)
• Local Data Quantity-Aware Weighted Averaging for Federated Learning with Dishonest Clients [32.11272564261484]
  Federated learning (FL) enables collaborative training of deep learning models without requiring data to leave local clients. Most commonly used aggregation method is weighted averaging based on the amount of data from each client, which is thought to reflect each client's contribution. We propose a novel secure underlineFederated underlineData qunderlineuantity-underlineaware weighted averaging method (FedDua) It enables FL servers to accurately predict the amount of training data from each client based on their local model gradients uploaded.
  arXiv  Detail & Related papers  (2025-04-17T01:50:24Z)
• FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated Clients [25.847042398060616]
  Federated Learning (FL) facilitates collaborative training of a shared global model without exposing clients' private data. We propose FedConv, a client-friendly FL framework, which minimizes the computation and memory burden on resource-constrained clients. We show that FedConv outperforms state-of-the-art FL systems in terms of model accuracy, computation and communication overhead.
  arXiv  Detail & Related papers  (2025-02-28T01:39:53Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• Stochastic Approximation Approach to Federated Machine Learning [0.0]
  This paper examines Federated learning (FL) in a Approximation (SA) framework. FL is a collaborative way to train neural network models across various participants or clients. It is observed that the proposed algorithm is robust and gives more reliable estimates of the weights.
  arXiv  Detail & Related papers  (2024-02-20T12:00:25Z)
• Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes [54.18186259484828]
  In Federated Learning (FL) paradigm, a parameter server (PS) concurrently communicates with distributed participating clients for model collection, update aggregation, and model distribution over multiple rounds. We show strong evidences that variable-length is beneficial for compression in FL. We present Fed-CVLC (Federated Learning Compression with Variable-Length Codes), which fine-tunes the code length in response to the dynamics of model updates.
  arXiv  Detail & Related papers  (2024-02-06T07:25:21Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Complement Sparsification: Low-Overhead Model Pruning for Federated Learning [2.0428960719376166]
  Federated Learning (FL) is a privacy-preserving distributed deep learning paradigm that involves substantial communication and computation effort. Existing model pruning/sparsification solutions cannot satisfy the requirements for low bidirectional communication overhead between the server and the clients. We propose Complement Sparsification (CS), a pruning mechanism that satisfies all these requirements through a complementary and collaborative pruning done at the server and the clients.
  arXiv  Detail & Related papers  (2023-03-10T23:07:02Z)
• FedCliP: Federated Learning with Client Pruning [3.796320380104124]
  Federated learning (FL) is a newly emerging distributed learning paradigm. One fundamental bottleneck in FL is the heavy communication overheads between the distributed clients and the central server. We propose FedCliP, the first communication efficient FL training framework from a macro perspective.
  arXiv  Detail & Related papers  (2023-01-17T09:15:37Z)
• ResFed: Communication Efficient Federated Learning by Transmitting Deep Compressed Residuals [24.13593410107805]
  Federated learning enables cooperative training among massively distributed clients by sharing their learned local model parameters. We introduce a residual-based federated learning framework (ResFed), where residuals rather than model parameters are transmitted in communication networks for training. By employing a common prediction rule, both locally and globally updated models are always fully recoverable in clients and the server.
  arXiv  Detail & Related papers  (2022-12-11T20:34:52Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• Over-the-Air Federated Learning from Heterogeneous Data [107.05618009955094]
  Federated learning (FL) is a framework for distributed learning of centralized models. We develop a Convergent OTA FL (COTAF) algorithm which enhances the common local gradient descent (SGD) FL algorithm. We numerically show that the precoding induced by COTAF notably improves the convergence rate and the accuracy of models trained via OTA FL.
  arXiv  Detail & Related papers  (2020-09-27T08:28:25Z)
• Distilled One-Shot Federated Learning [13.294757670979031]
  We propose Distilled One-Shot Federated Learning (DOSFL) to significantly reduce the communication cost while achieving comparable performance. In just one round, each client distills their private dataset, sends the synthetic data (e.g. images or sentences) to the server, and collectively trains a global model. With this weight-less and gradient-less design, the total communication cost of DOSFL is up to three orders of magnitude less than FedAvg.
  arXiv  Detail & Related papers  (2020-09-17T01:14:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.