Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning

• URL: http://arxiv.org/abs/2606.01128v1
• Date: Sun, 31 May 2026 10:02:15 GMT
• Title: Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning
• Abstract summary: Local MixVR is a distributed framework that integrates local updates with variance-reduction techniques to mitigate local noise.<n>In common regimes where $MOleft(N1/4right)$, Local MixVR outperforms the state-of-the-art Minibatch Accelerated SGD baseline.
• Score: 47.950033101660644
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Communication overhead is a crucial bottleneck in scalable distributed learning. While existing methods aim to efficiently utilize data points, such as Local SGD, Minibatch SGD, and their accelerated variants, they still exhibit communication-round complexity that scales with the total number of samples $N$. In this paper, we introduce Local MixVR, a distributed framework that integrates local updates with variance-reduction techniques to mitigate local noise. We show that Local MixVR is the first distributed method to eliminate the dependence of communication complexity on $N$, achieving a complexity that scales only with the number of workers $M$. In common regimes where $M<O\left(N^{1/4}\right)$, Local MixVR outperforms the state-of-the-art Minibatch Accelerated SGD baseline, bridging a long-standing gap in distributed optimization and establishing a new paradigm for communication-efficient training.

Related papers

• Unifying Local Communications and Local Updates for LLM Pretraining [22.321729036377846]
  Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters.<n>We introduce GASLoC, a novel decentralized pre-training algorithm that generalizes the notion of communication acceleration to allow a practical gossip-based training framework.<n>We demonstrate that GASLoC outperforms state-of-the-art decentralized algorithms in single step per communication setting for a number of topologies.
  arXiv  Detail & Related papers  (2026-06-09T16:40:54Z)
• Distributed Machine Learning Approach for Low-Latency Localization in Cell-Free Massive MIMO Systems [16.842941544015194]
  Low-latency localization is critical in cellular networks to support real-time applications requiring precise positioning.<n>We propose a distributed machine learning framework for fingerprint-based localization tailored to cell-free massive multiple-input multiple-output (MIMO) systems.
  arXiv  Detail & Related papers  (2025-07-16T06:05:16Z)
• A One-shot Framework for Distributed Clustered Learning in Heterogeneous Environments [54.172993875654015]
  The paper proposes a family of communication efficient methods for distributed learning in heterogeneous environments. One-shot approach, based on local computations at the users and a clustering based aggregation step at the server is shown to provide strong learning guarantees. For strongly convex problems it is shown that, as long as the number of data points per user is above a threshold, the proposed approach achieves order-optimal mean-squared error rates in terms of the sample size.
  arXiv  Detail & Related papers  (2022-09-22T09:04:10Z)
• Federated Minimax Optimization: Improved Convergence Analyses and Algorithms [32.062312674333775]
  We consider non minimax optimization, is gaining prominence many modern machine learning applications such as GANs. We provide a novel and tighter analysis algorithm, improves convergence communication guarantees in the existing literature.
  arXiv  Detail & Related papers  (2022-03-09T16:21:31Z)
• Escaping Saddle Points with Bias-Variance Reduced Local Perturbed SGD for Communication Efficient Nonconvex Distributed Learning [58.79085525115987]
  Local methods are one of the promising approaches to reduce communication time. We show that the communication complexity is better than non-local methods when the local datasets is smaller than the smoothness local loss.
  arXiv  Detail & Related papers  (2022-02-12T15:12:17Z)
• FedMix: Approximation of Mixup under Mean Augmented Federated Learning [60.503258658382]
  Federated learning (FL) allows edge devices to collectively learn a model without directly sharing data within each device. Current state-of-the-art algorithms suffer from performance degradation as the heterogeneity of local data across clients increases. We propose a new augmentation algorithm, named FedMix, which is inspired by a phenomenal yet simple data augmentation method, Mixup.
  arXiv  Detail & Related papers  (2021-07-01T06:14:51Z)
• Clustered Federated Learning via Generalized Total Variation Minimization [83.26141667853057]
  We study optimization methods to train local (or personalized) models for local datasets with a decentralized network structure. Our main conceptual contribution is to formulate federated learning as total variation minimization (GTV) Our main algorithmic contribution is a fully decentralized federated learning algorithm.
  arXiv  Detail & Related papers  (2021-05-26T18:07:19Z)
• Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiency [15.04034188283642]
  Local SGD is a promising approach to overcome the communication overhead in distributed learning. We show that local SGDA can provably optimize distributed minimax problems in both homogeneous and heterogeneous data.
  arXiv  Detail & Related papers  (2021-02-25T20:15:18Z)
• Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning [46.32232395989181]
  We aim at learning local efficiently in terms of communication and computational complexity. One of the important learning scenarios in distributed learning is the Federated Learning scenario.
  arXiv  Detail & Related papers  (2021-02-05T14:32:28Z)
• Coded Stochastic ADMM for Decentralized Consensus Optimization with Edge Computing [113.52575069030192]
  Big data, including applications with high security requirements, are often collected and stored on multiple heterogeneous devices, such as mobile devices, drones and vehicles. Due to the limitations of communication costs and security requirements, it is of paramount importance to extract information in a decentralized manner instead of aggregating data to a fusion center. We consider the problem of learning model parameters in a multi-agent system with data locally processed via distributed edge nodes. A class of mini-batch alternating direction method of multipliers (ADMM) algorithms is explored to develop the distributed learning model.
  arXiv  Detail & Related papers  (2020-10-02T10:41:59Z)
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates [70.9701218475002]
  We introduce a unified convergence analysis of decentralized communication methods. We derive universal convergence rates for several applications. Our proofs rely on weak assumptions.
  arXiv  Detail & Related papers  (2020-03-23T17:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.