Decentralized SGD with Over-the-Air Computation

• URL: http://arxiv.org/abs/2003.04216v1
• Date: Fri, 6 Mar 2020 15:33:59 GMT
• Title: Decentralized SGD with Over-the-Air Computation
• Authors: Emre Ozfatura, Stefano Rini, Deniz Gunduz
• Abstract summary: We study the performance of decentralized numerically gradient descent (DSGD) in a wireless network. We assume that transmissions are prone to additive noise and interference. We show that the OAC-MAC scheme attains better convergence performance with a fewer communication rounds.
• Score: 13.159777131162961
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the performance of decentralized stochastic gradient descent (DSGD) in a wireless network, where the nodes collaboratively optimize an objective function using their local datasets. Unlike the conventional setting, where the nodes communicate over error-free orthogonal communication links, we assume that transmissions are prone to additive noise and interference.We first consider a point-to-point (P2P) transmission strategy, termed the OAC-P2P scheme, in which the node pairs are scheduled in an orthogonal fashion to minimize interference. Since in the DSGD framework, each node requires a linear combination of the neighboring models at the consensus step, we then propose the OAC-MAC scheme, which utilizes the signal superposition property of the wireless medium to achieve over-the-air computation (OAC). For both schemes, we cast the scheduling problem as a graph coloring problem. We numerically evaluate the performance of these two schemes for the MNIST image classification task under various network conditions. We show that the OAC-MAC scheme attains better convergence performance with a fewer communication rounds.

Related papers

• FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression [55.992528247880685]
  Decentralized training faces significant challenges regarding system design and efficiency. We present FusionLLM, a decentralized training system designed and implemented for training large deep neural networks (DNNs) We show that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.
  arXiv  Detail & Related papers  (2024-10-16T16:13:19Z)
• Unsupervised Graph-based Learning Method for Sub-band Allocation in 6G Subnetworks [2.0583251142940377]
  We present an unsupervised approach for frequency sub-band allocation in wireless networks using graph-based learning. We model the subnetwork deployment as a conflict graph and propose an unsupervised learning approach inspired by the graph colouring and the Potts model to optimize the sub-band allocation.
  arXiv  Detail & Related papers  (2023-12-13T12:57:55Z)
• Wireless Federated $k$-Means Clustering with Non-coherent Over-the-Air Computation [14.087062902871212]
  OAC scheme relies on an encoder exploiting the representation of a number in a balanced number system. Reinitialization method for ineffectively used centroids is proposed to improve the performance of the proposed method for heterogeneous data distribution.
  arXiv  Detail & Related papers  (2023-08-11T20:12:26Z)
• Non-Coherent Over-the-Air Decentralized Gradient Descent [0.0]
  Implementing Decentralized Gradient Descent in wireless systems is challenging due to noise, fading, and limited bandwidth. This paper introduces a scalable DGD algorithm that eliminates the need for scheduling, topology information, or CSI.
  arXiv  Detail & Related papers  (2022-11-19T19:15:34Z)
• Bandwidth-efficient distributed neural network architectures with application to body sensor networks [73.02174868813475]
  This paper describes a conceptual design methodology to design distributed neural network architectures. We show that the proposed framework enables up to a factor 20 in bandwidth reduction with minimal loss. While the application focus of this paper is on wearable brain-computer interfaces, the proposed methodology can be applied in other sensor network-like applications as well.
  arXiv  Detail & Related papers  (2022-10-14T12:35:32Z)
• Interpolation-based Correlation Reduction Network for Semi-Supervised Graph Learning [49.94816548023729]
  We propose a novel graph contrastive learning method, termed Interpolation-based Correlation Reduction Network (ICRN) In our method, we improve the discriminative capability of the latent feature by enlarging the margin of decision boundaries. By combining the two settings, we extract rich supervision information from both the abundant unlabeled nodes and the rare yet valuable labeled nodes for discnative representation learning.
  arXiv  Detail & Related papers  (2022-06-06T14:26:34Z)
• Federated Learning for Energy-limited Wireless Networks: A Partial Model Aggregation Approach [79.59560136273917]
  limited communication resources, bandwidth and energy, and data heterogeneity across devices are main bottlenecks for federated learning (FL) We first devise a novel FL framework with partial model aggregation (PMA) The proposed PMA-FL improves 2.72% and 11.6% accuracy on two typical heterogeneous datasets.
  arXiv  Detail & Related papers  (2022-04-20T19:09:52Z)
• Learning Resilient Radio Resource Management Policies with Graph Neural Networks [124.89036526192268]
  We formulate a resilient radio resource management problem with per-user minimum-capacity constraints. We show that we can parameterize the user selection and power control policies using a finite set of parameters. Thanks to such adaptation, our proposed method achieves a superior tradeoff between the average rate and the 5th percentile rate.
  arXiv  Detail & Related papers  (2022-03-07T19:40:39Z)
• Graph-based Algorithm Unfolding for Energy-aware Power Allocation in Wireless Networks [27.600081147252155]
  We develop a novel graph sumable framework to maximize energy efficiency in wireless communication networks. We show the permutation training which is a desirable property for models of wireless network data. Results demonstrate its generalizability across different network topologies.
  arXiv  Detail & Related papers  (2022-01-27T20:23:24Z)
• Over-the-Air Decentralized Federated Learning [28.593149477080605]
  We consider decentralized federated learning (FL) over wireless networks, where over-the-air computation (AirComp) is adopted to facilitate the local model consensus in a device-to-device (D2D) communication manner. We propose an AirComp-based DSGD with gradient tracking and variance reduction (DSGT-VR) algorithm, where both precoding and decoding strategies are developed for D2D communication. We prove that the proposed algorithm converges linearly and establish the optimality gap for strongly convex and smooth loss functions, taking into account the channel fading and noise.
  arXiv  Detail & Related papers  (2021-06-15T09:42:33Z)
• Wireless Power Control via Counterfactual Optimization of Graph Neural Networks [124.89036526192268]
  We consider the problem of downlink power control in wireless networks, consisting of multiple transmitter-receiver pairs communicating over a single shared wireless medium. To mitigate the interference among concurrent transmissions, we leverage the network topology to create a graph neural network architecture. We then use an unsupervised primal-dual counterfactual optimization approach to learn optimal power allocation decisions.
  arXiv  Detail & Related papers  (2020-02-17T07:54:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.