FedScalar: A Communication efficient Federated Learning

• URL: http://arxiv.org/abs/2410.02260v1
• Date: Thu, 3 Oct 2024 07:06:49 GMT
• Title: FedScalar: A Communication efficient Federated Learning
• Authors: M. Rostami, S. S. Kia,
• Abstract summary: Federated learning (FL) has gained considerable popularity for distributed machine learning. emphFedScalar enables agents to communicate updates using a single scalar.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) has gained considerable popularity for distributed machine learning due to its ability to preserve the privacy of participating agents by eliminating the need for data aggregation. Nevertheless, communication costs between agents and the central server in FL are substantial in large-scale problems and remain a limiting factor for this algorithm. This paper introduces an innovative algorithm, called \emph{FedScalar}, within the federated learning framework aimed at improving communication efficiency. Unlike traditional FL methods that require agents to send high-dimensional vectors to the server, \emph{FedScalar} enables agents to communicate updates using a single scalar. Each agent encodes its updated model parameters into a scalar through the inner product between its local update difference and a random vector, which is then transmitted to the server. The server decodes this information by projecting the averaged scalar values onto the random vector. Our method thereby significantly reduces communication overhead. Technically, we demonstrate that the proposed algorithm achieves a convergence rate of $O(1/\sqrt{K})$ to a stationary point for smooth, non-convex loss functions. Additionally, our analysis shows that altering the underlying distribution of the random vector generated by the server can reduce the variance during the aggregation step of the algorithm. Finally, we validate the performance and communication efficiency of our algorithm with numerical simulations.

Related papers

• Noise-Robust and Resource-Efficient ADMM-based Federated Learning [6.957420925496431]
  Federated learning (FL) leverages client-server communications to train global models on decentralized data. We propose a novel FL algorithm that enhances robustness against communication noise while also reducing communication load.
  arXiv  Detail & Related papers  (2024-09-20T12:32:22Z)
• SignSGD with Federated Voting [69.06621279967865]
  SignSGD with majority voting (signSGD-MV) is an effective distributed learning algorithm that can significantly reduce communication costs by one-bit quantization. We propose a novel signSGD with textitfederated voting (signSGD-FV) The idea of federated voting is to exploit learnable weights to perform weighted majority voting. We demonstrate that the proposed signSGD-FV algorithm has a theoretical convergence guarantee even when edge devices use heterogeneous mini-batch sizes.
  arXiv  Detail & Related papers  (2024-03-25T02:32:43Z)
• Communication Efficient ConFederated Learning: An Event-Triggered SAGA Approach [67.27031215756121]
  Federated learning (FL) is a machine learning paradigm that targets model training without gathering the local data over various data sources. Standard FL, which employs a single server, can only support a limited number of users, leading to degraded learning capability. In this work, we consider a multi-server FL framework, referred to as emphConfederated Learning (CFL) in order to accommodate a larger number of users.
  arXiv  Detail & Related papers  (2024-02-28T03:27:10Z)
• Asynchronous Local Computations in Distributed Bayesian Learning [8.516532665507835]
  We propose gossip-based communication to leverage fast computations and reduce communication overhead simultaneously. We observe faster initial convergence and improved performance accuracy, especially in the low data range. We achieve on average 78% and over 90% classification accuracy respectively on the Gamma Telescope and mHealth data sets from the UCI ML repository.
  arXiv  Detail & Related papers  (2023-11-06T20:11:41Z)
• OFedQIT: Communication-Efficient Online Federated Learning via Quantization and Intermittent Transmission [7.6058140480517356]
  Online federated learning (OFL) is a promising framework to collaboratively learn a sequence of non-linear functions (or models) from distributed streaming data. We propose a communication-efficient OFL algorithm (named OFedQIT) by means of a quantization and an intermittent transmission. Our analysis reveals that OFedQIT successfully addresses the drawbacks of OFedAvg while maintaining superior learning accuracy.
  arXiv  Detail & Related papers  (2022-05-13T07:46:43Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• A Newton-type algorithm for federated learning based on incremental Hessian eigenvector sharing [5.404315085380945]
  We present an original communication-constrained Newton-type (NT) algorithm designed to accelerate Federated Learning (FL) The proposed solution is thoroughly validated on real datasets.
  arXiv  Detail & Related papers  (2022-02-11T17:52:56Z)
• Faster Non-Convex Federated Learning via Global and Local Momentum [57.52663209739171]
  textttFedGLOMO is the first (first-order) FLtexttFedGLOMO algorithm. Our algorithm is provably optimal even with communication between the clients and the server.
  arXiv  Detail & Related papers  (2020-12-07T21:05:31Z)
• Coded Stochastic ADMM for Decentralized Consensus Optimization with Edge Computing [113.52575069030192]
  Big data, including applications with high security requirements, are often collected and stored on multiple heterogeneous devices, such as mobile devices, drones and vehicles. Due to the limitations of communication costs and security requirements, it is of paramount importance to extract information in a decentralized manner instead of aggregating data to a fusion center. We consider the problem of learning model parameters in a multi-agent system with data locally processed via distributed edge nodes. A class of mini-batch alternating direction method of multipliers (ADMM) algorithms is explored to develop the distributed learning model.
  arXiv  Detail & Related papers  (2020-10-02T10:41:59Z)
• Communication-Efficient Robust Federated Learning Over Heterogeneous Datasets [147.11434031193164]
  This work investigates fault-resilient federated learning when the data samples are non-uniformly distributed across workers. In the presence of adversarially faulty workers who may strategically corrupt datasets, the local messages exchanged can be unreliable. The present work introduces a fault-resilient gradient (FRPG) algorithm that relies on Nesterov's acceleration technique. For strongly convex loss functions, FRPG and LFRPG have provably faster convergence rates than a benchmark robust aggregation algorithm.
  arXiv  Detail & Related papers  (2020-06-17T16:50:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.