Fed-ZOE: Communication-Efficient Over-the-Air Federated Learning via Zeroth-Order Estimation

• URL: http://arxiv.org/abs/2412.16779v1
• Date: Sat, 21 Dec 2024 21:24:58 GMT
• Title: Fed-ZOE: Communication-Efficient Over-the-Air Federated Learning via Zeroth-Order Estimation
• Authors: Jonggyu Jang, Hyeonsu Lyu, David J. Love, Hyun Jong Yang,
• Abstract summary: Fed-ZOE is an efficient framework inspired by the randomized gradient estimator (RGE) commonly used in zeroth-order optimization (ZOO) Fed-ZOE achieves performance comparable to Fed-OtA while drastically reducing communication costs.
• Score: 15.026407830543086
• License:
• Abstract: As 6G and beyond networks grow increasingly complex and interconnected, federated learning (FL) emerges as an indispensable paradigm for securely and efficiently leveraging decentralized edge data for AI. By virtue of the superposition property of communication signals, over-the-air FL (OtA-FL) achieves constant communication overhead irrespective of the number of edge devices (EDs). However, training neural networks over the air still incurs substantial communication costs, as the number of transmitted symbols equals the number of trainable parameters. To alleviate this issue, the most straightforward approach is to reduce the number of transmitted symbols by 1) gradient compression and 2) gradient sparsification. Unfortunately, these methods are incompatible with OtA-FL due to the loss of its superposition property. In this work, we introduce federated zeroth-order estimation (Fed-ZOE), an efficient framework inspired by the randomized gradient estimator (RGE) commonly used in zeroth-order optimization (ZOO). In FedZOE, EDs perform local weight updates as in standard FL, but instead of transmitting full gradient vectors, they send compressed local model update vectors in the form of several scalar-valued inner products between the local model update vectors and random vectors. These scalar values enable the parameter server (PS) to reconstruct the gradient using the RGE trick with highly reduced overhead, as well as preserving the superposition property. Unlike conventional ZOO leveraging RGE for step-wise gradient descent, Fed-ZOE compresses local model update vectors before transmission, thereby achieving higher accuracy and computational efficiency. Numerical evaluations using ResNet-18 on datasets such as CIFAR-10, TinyImageNet, SVHN, CIFAR-100, and Brain-CT demonstrate that Fed-ZOE achieves performance comparable to Fed-OtA while drastically reducing communication costs.

Related papers

• Adaptive Quantization Resolution and Power Control for Federated Learning over Cell-free Networks [41.23236059700041]
  Federated learning (FL) is a distributed learning framework where users train a global model by exchanging local model updates with a server instead of raw datasets. Cell-free massive multiple-input multipleoutput (CFmMIMO) is a promising solution to serve numerous users on the same time/frequency resource with similar rates. In this paper, we co-optimize the physical layer with the FL application to mitigate the straggler effect.
  arXiv  Detail & Related papers  (2024-12-14T16:08:05Z)
• FedScalar: A Communication efficient Federated Learning [0.0]
  Federated learning (FL) has gained considerable popularity for distributed machine learning. emphFedScalar enables agents to communicate updates using a single scalar.
  arXiv  Detail & Related papers  (2024-10-03T07:06:49Z)
• Adaptive Federated Pruning in Hierarchical Wireless Networks [69.6417645730093]
  Federated Learning (FL) is a privacy-preserving distributed learning framework where a server aggregates models updated by multiple devices without accessing their private datasets. In this paper, we introduce model pruning for HFL in wireless networks to reduce the neural network scale. We show that our proposed HFL with model pruning achieves similar learning accuracy compared with the HFL without model pruning and reduces about 50 percent communication cost.
  arXiv  Detail & Related papers  (2023-05-15T22:04:49Z)
• Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning [28.65635956111857]
  Federated learning (FL) enables edge devices to collaboratively train a global model without sharing their private data. To enhance the training efficiency of FL, various algorithms have been proposed, ranging from first-order computation to first-order methods.
  arXiv  Detail & Related papers  (2022-01-24T08:56:06Z)
• Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better [88.28293442298015]
  Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. We develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network.
  arXiv  Detail & Related papers  (2021-12-18T02:26:38Z)
• Wireless Federated Learning with Limited Communication and Differential Privacy [21.328507360172203]
  This paper investigates the role of dimensionality reduction in efficient communication and differential privacy (DP) of the local datasets at the remote users for over-the-air computation (AirComp)-based federated learning (FL) model.
  arXiv  Detail & Related papers  (2021-06-01T15:23:12Z)
• Rate Region for Indirect Multiterminal Source Coding in Federated Learning [49.574683687858126]
  A large number of edge devices send their updates to the edge at each round of the local model. Existing works do not leverage the focus in the information transmitted by different edges. This paper studies the rate region for the indirect multiterminal source coding FL.
  arXiv  Detail & Related papers  (2021-01-21T16:17:39Z)
• Bayesian Federated Learning over Wireless Networks [87.37301441859925]
  Federated learning is a privacy-preserving and distributed training method using heterogeneous data sets stored at local devices. This paper presents an efficient modified BFL algorithm called scalableBFL (SBFL)
  arXiv  Detail & Related papers  (2020-12-31T07:32:44Z)
• Over-the-Air Federated Learning from Heterogeneous Data [107.05618009955094]
  Federated learning (FL) is a framework for distributed learning of centralized models. We develop a Convergent OTA FL (COTAF) algorithm which enhances the common local gradient descent (SGD) FL algorithm. We numerically show that the precoding induced by COTAF notably improves the convergence rate and the accuracy of models trained via OTA FL.
  arXiv  Detail & Related papers  (2020-09-27T08:28:25Z)
• Gradient Statistics Aware Power Control for Over-the-Air Federated Learning [59.40860710441232]
  Federated learning (FL) is a promising technique that enables many edge devices to train a machine learning model collaboratively in wireless networks. This paper studies the power control problem for over-the-air FL by taking gradient statistics into account.
  arXiv  Detail & Related papers  (2020-03-04T14:06:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.