BICompFL: Stochastic Federated Learning with Bi-Directional Compression

• URL: http://arxiv.org/abs/2502.00206v1
• Date: Fri, 31 Jan 2025 22:48:43 GMT
• Title: BICompFL: Stochastic Federated Learning with Bi-Directional Compression
• Authors: Maximilian Egger, Rawad Bitar, Antonia Wachter-Zeh, Nir Weinberger, Deniz Gündüz,
• Abstract summary: We address the prominent communication bottleneck in federated learning (FL) We show that bi-directional compression for FL has inherent challenges, which we address by BICompFL. Our BICompFL is experimentally shown to reduce the communication cost by an order of magnitude compared to multiple benchmarks, while maintaining state-of-the-art accuracies.
• Score: 70.37629026426104
• License:
• Abstract: We address the prominent communication bottleneck in federated learning (FL). We specifically consider stochastic FL, in which models or compressed model updates are specified by distributions rather than deterministic parameters. Stochastic FL offers a principled approach to compression, and has been shown to reduce the communication load under perfect downlink transmission from the federator to the clients. However, in practice, both the uplink and downlink communications are constrained. We show that bi-directional compression for stochastic FL has inherent challenges, which we address by introducing BICompFL. Our BICompFL is experimentally shown to reduce the communication cost by an order of magnitude compared to multiple benchmarks, while maintaining state-of-the-art accuracies. Theoretically, we study the communication cost of BICompFL through a new analysis of an importance-sampling based technique, which exposes the interplay between uplink and downlink communication costs.

Related papers

• Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization [15.73877955614998]
  This paper presents a novel communication algorithm - DeComFL - which reduces the communication cost from $mathscrO(d)$ to $mathscrO(d)$ by transmitting only a constant number of scalar values between clients. Empirical evaluations, encompassing both classic deep learning training and large language model fine-tuning, demonstrate significant reductions in communication overhead.
  arXiv  Detail & Related papers  (2024-05-24T18:07:05Z)
• FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models [56.21666819468249]
  Federated Learning (FL) has garnered increasing attention due to its unique characteristic of allowing heterogeneous clients to process their private data locally and interact with a central server. We introduce FedComLoc, integrating practical and effective compression into emphScaffnew to further enhance communication efficiency.
  arXiv  Detail & Related papers  (2024-03-14T22:29:59Z)
• Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes [54.18186259484828]
  In Federated Learning (FL) paradigm, a parameter server (PS) concurrently communicates with distributed participating clients for model collection, update aggregation, and model distribution over multiple rounds. We show strong evidences that variable-length is beneficial for compression in FL. We present Fed-CVLC (Federated Learning Compression with Variable-Length Codes), which fine-tunes the code length in response to the dynamics of model updates.
  arXiv  Detail & Related papers  (2024-02-06T07:25:21Z)
• How Robust is Federated Learning to Communication Error? A Comparison Study Between Uplink and Downlink Channels [13.885735785986164]
  This paper investigates the robustness of federated learning to the uplink and downlink communication error. It is shown that the uplink communication in FL can tolerate a higher bit error rate (BER) than downlink communication.
  arXiv  Detail & Related papers  (2023-10-25T14:03:11Z)
• Semi-Federated Learning: Convergence Analysis and Optimization of A Hybrid Learning Framework [70.83511997272457]
  We propose a semi-federated learning (SemiFL) paradigm to leverage both the base station (BS) and devices for a hybrid implementation of centralized learning (CL) and FL. We propose a two-stage algorithm to solve this intractable problem, in which we provide the closed-form solutions to the beamformers.
  arXiv  Detail & Related papers  (2023-10-04T03:32:39Z)
• Stochastic Controlled Averaging for Federated Learning with Communication Compression [36.09135695005069]
  We propose two compressed FL algorithms, SCALLION and SCAFCOM, to support unbiased and biased compression. Both the proposed methods outperform the existing compressed FL methods in terms of communication and computation complexities.
  arXiv  Detail & Related papers  (2023-08-16T06:35:36Z)
• Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noise [10.685862129925729]
  We propose an improved convergence analysis technique that characterizes the distributed learning paradigm with imperfect/noisy uplink and downlink communications. Such imperfect communication scenarios arise in the practical deployment of FL in emerging communication systems and protocols.
  arXiv  Detail & Related papers  (2023-07-14T15:35:57Z)
• Fundamental Limits of Communication Efficiency for Model Aggregation in Distributed Learning: A Rate-Distortion Approach [54.311495894129585]
  We study the limit of communication cost of model aggregation in distributed learning from a rate-distortion perspective. It is found that the communication gain by exploiting the correlation between worker nodes is significant for SignSGD.
  arXiv  Detail & Related papers  (2022-06-28T13:10:40Z)
• Delay Minimization for Federated Learning Over Wireless Communication Networks [172.42768672943365]
  The problem of delay computation for federated learning (FL) over wireless communication networks is investigated. A bisection search algorithm is proposed to obtain the optimal solution. Simulation results show that the proposed algorithm can reduce delay by up to 27.3% compared to conventional FL methods.
  arXiv  Detail & Related papers  (2020-07-05T19:00:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.