Optimal Data Splitting in Distributed Optimization for Machine Learning

• URL: http://arxiv.org/abs/2401.07809v2
• Date: Tue, 26 Mar 2024 17:29:07 GMT
• Title: Optimal Data Splitting in Distributed Optimization for Machine Learning
• Authors: Daniil Medyakov, Gleb Molodtsov, Aleksandr Beznosikov, Alexander Gasnikov,
• Abstract summary: This study focuses on an optimal ratio of distributed data between the server and local machines for any costs of communications and local computations. The running times of the network are compared between uniform and optimal distributions. The superior theoretical performance of our solutions is experimentally validated.
• Score: 85.99744701008802
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The distributed optimization problem has become increasingly relevant recently. It has a lot of advantages such as processing a large amount of data in less time compared to non-distributed methods. However, most distributed approaches suffer from a significant bottleneck - the cost of communications. Therefore, a large amount of research has recently been directed at solving this problem. One such approach uses local data similarity. In particular, there exists an algorithm provably optimally exploiting the similarity property. But this result, as well as results from other works solve the communication bottleneck by focusing only on the fact that communication is significantly more expensive than local computing and does not take into account the various capacities of network devices and the different relationship between communication time and local computing expenses. We consider this setup and the objective of this study is to achieve an optimal ratio of distributed data between the server and local machines for any costs of communications and local computations. The running times of the network are compared between uniform and optimal distributions. The superior theoretical performance of our solutions is experimentally validated.

Related papers

• Fast networked data selection via distributed smoothed quantile estimation [6.002041236376175]
  We establish a connection between selecting the most informative data and finding the top-$k$ elements of a multiset. The top-$k$ selection in a network can be formulated as a distributed nonsmooth convex optimization problem known as quantile estimation. We characterize the complexity required to achieve top-$k$ selection, a challenging task due to the lack of strong convexity.
  arXiv  Detail & Related papers  (2024-06-04T03:26:15Z)
• Optimizing the Optimal Weighted Average: Efficient Distributed Sparse Classification [50.406127962933915]
  ACOWA allows an extra round of communication to achieve noticeably better approximation quality with minor runtime increases. Results show that ACOWA obtains solutions that are more faithful to the empirical risk minimizer and attain substantially higher accuracy than other distributed algorithms.
  arXiv  Detail & Related papers  (2024-06-03T19:43:06Z)
• TAMUNA: Doubly Accelerated Distributed Optimization with Local Training, Compression, and Partial Participation [53.84175614198885]
  In distributed optimization and learning, several machines alternate between local computations in parallel and communication with a distant server. We propose TAMUNA, the first algorithm for distributed optimization that leveraged the two strategies of local training and compression jointly and allows for partial participation.
  arXiv  Detail & Related papers  (2023-02-20T08:37:44Z)
• Communication-Efficient Adam-Type Algorithms for Distributed Data Mining [93.50424502011626]
  We propose a class of novel distributed Adam-type algorithms (emphi.e., SketchedAMSGrad) utilizing sketching. Our new algorithm achieves a fast convergence rate of $O(frac1sqrtnT + frac1(k/d)2 T)$ with the communication cost of $O(k log(d))$ at each iteration.
  arXiv  Detail & Related papers  (2022-10-14T01:42:05Z)
• Collaborative Learning over Wireless Networks: An Introductory Overview [84.09366153693361]
  We will mainly focus on collaborative training across wireless devices. Many distributed optimization algorithms have been developed over the last decades. They provide data locality; that is, a joint model can be trained collaboratively while the data available at each participating device remains local.
  arXiv  Detail & Related papers  (2021-12-07T20:15:39Z)
• Acceleration in Distributed Optimization Under Similarity [72.54787082152278]
  We study distributed (strongly convex) optimization problems over a network of agents, with no centralized nodes. An $varepsilon$-solution is achieved in $tildemathcalrhoObig(sqrtfracbeta/mu (1-)log1/varepsilonbig)$ number of communications steps. This rate matches (up to poly-log factors) for the first time lower complexity communication bounds of distributed gossip-algorithms applied to the class of problems of interest.
  arXiv  Detail & Related papers  (2021-10-24T04:03:00Z)
• Distributed Optimization, Averaging via ADMM, and Network Topology [0.0]
  We study the connection between network topology and convergence rates for different algorithms on a real world problem of sensor localization. We also show interesting connections between ADMM and lifted Markov chains besides providing an explicitly characterization of its convergence.
  arXiv  Detail & Related papers  (2020-09-05T21:44:39Z)
• Communication-efficient distributed eigenspace estimation [31.69089186688224]
  We develop a communication-efficient distributed algorithm for computing the leading invariant subspace of a data matrix. Our algorithm uses a novel alignment scheme that minimizes the Procrustean distance between local solutions and a reference solution. We show that our algorithm achieves a similar error rate to that of a centralized estimator.
  arXiv  Detail & Related papers  (2020-09-05T02:11:22Z)
• Scaling-up Distributed Processing of Data Streams for Machine Learning [10.581140430698103]
  This paper reviews recently developed methods that focus on large-scale distributed optimization in the compute- and bandwidth-limited regime. It focuses on methods that solve: (i) distributed convex problems, and (ii) distributed principal component analysis, which is a non problem with geometric structure that permits global convergence.
  arXiv  Detail & Related papers  (2020-05-18T16:28:54Z)
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey [22.42450750097714]
  We provide a comprehensive survey of the communication-efficient distributed training algorithms. We first propose a taxonomy of data-parallel distributed training algorithms. We then investigate state-of-the-art studies that address problems in these four dimensions.
  arXiv  Detail & Related papers  (2020-03-10T05:42:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.