Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo

• URL: http://arxiv.org/abs/2606.22768v1
• Date: Mon, 22 Jun 2026 02:15:13 GMT
• Title: Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
• Abstract summary: DiLoCo's outer synchronization remains bandwidth-heavy and brittle to stragglers and transient failures.<n>We relax exact synchronization to approximate synchronization via mixing/gossip, which degrades gracefully under delays and communication failures.<n>Our framework substantially improves compute utilization compared to DiLoCo, with training progress ranging from comparable to closely matching it, and is more robust to failures.
• Score: 53.37011208698893
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently, its outer synchronization remains bandwidth-heavy and brittle to stragglers and transient failures. We relax exact synchronization to approximate synchronization via mixing/gossip, which degrades gracefully under delays and communication failures. This allows us to factorize DiLoCo synchronization into a non-blocking mixing step that overlaps computation with no staleness, and a blocking mixing step that tightens worker agreement, yielding a tunable trade-off between compute utilization and optimization stability. On up to billion-parameter language models in low-bandwidth settings, our framework substantially improves compute utilization compared to DiLoCo, with training progress ranging from comparable to closely matching it, and is more robust to failures.

Related papers

• SparseRL-Sync: Lossless Weight Synchronization with ~100x Less Communication [0.6775438671102701]
  We propose and implement SparseRL-Sync, which replaces full-weight transfers with a sparse update payload.<n>SparseRL-Sync significantly improves scalability and end-to-end efficiency in bandwidth-limited and highly asynchronous RL settings.
  arXiv  Detail & Related papers  (2026-05-08T06:34:51Z)
• Decoupled DiLoCo for Resilient Distributed Pre-training [19.201912399584813]
  We introduce Decoupled DiLoCo, an evolution of the DiLoCo framework designed to break the lock-step synchronization barrier.<n>We achieve significantly improved training efficiency in failure-prone environments with millions of simulated chips with strictly zero global downtime.
  arXiv  Detail & Related papers  (2026-04-23T08:45:38Z)
• Asynchronous Heavy-Tailed Optimization [23.148100548141983]
  We propose two communication schemes that handle stragglers with asynchronous updates in the presence of heavy-tailed gradient noise.<n>Our convergence guarantees under heavy-tailed noise match the rate of the synchronous counterparts and improve delay tolerance compared with existing asynchronous approaches.
  arXiv  Detail & Related papers  (2026-02-20T05:28:48Z)
• AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism [54.8494905524997]
  We introduce asynchronous updates across both parallelism axes, relaxing the co-location requirement.<n>We provide convergence guarantees for both sparse averaging and asynchronous updates.<n>Experiments on large-scale language models demonstrate that our approach matches the performance of the fully synchronous baseline.
  arXiv  Detail & Related papers  (2026-01-30T01:24:47Z)
• AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models [34.54482364155804]
  We propose a three-stage method that combines Multi-Instance Training (MIT), Adaptive Batched DiLoCo, and switch mode mechanism.<n>MIT allows individual nodes to run multiple lightweight training streams with different model instances in parallel.<n> Adaptive Batched DiLoCo dynamically adjusts local batch sizes to balance computation and communication.<n>Switch mode seamlessly introduces gradient accumulation once adaptive batch sizes grow beyond hardware-friendly limits.
  arXiv  Detail & Related papers  (2025-08-25T16:35:57Z)
• A Novel Collaborative Framework for Efficient Synchronization in Split Federated Learning over Wireless Networks [4.462403784684656]
  We propose a new framework, called Collaborative Split Federated Learning (CSFL), that redefines workload redistribution through device-to-device collaboration.<n>CSFL enables efficient devices, after completing their own forward propagation, to seamlessly take over the unfinished layers of bottleneck devices.<n>This collaborative process, supported by D2D communications, allows bottleneck devices to offload earlier while maintaining synchronized progression across the network.
  arXiv  Detail & Related papers  (2025-03-18T22:11:54Z)
• Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch [66.84195842685459]
  Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time.<n>Recently, distributed algorithms like DiLoCo have relaxed such co-location constraint.<n>We show experimentally that we can distribute training of billion-scale parameters and reach similar quality as before.
  arXiv  Detail & Related papers  (2025-01-30T17:23:50Z)
• Digital Twin-Assisted Federated Learning with Blockchain in Multi-tier Computing Systems [67.14406100332671]
  In Industry 4.0 systems, resource-constrained edge devices engage in frequent data interactions. This paper proposes a digital twin (DT) and federated digital twin (FL) scheme. The efficacy of our proposed cooperative interference-based FL process has been verified through numerical analysis.
  arXiv  Detail & Related papers  (2024-11-04T17:48:02Z)
• LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression [56.01900711954956]
  We introduce LoCoDL, a communication-efficient algorithm that leverages the two popular and effective techniques of Local training, which reduces the communication frequency, and Compression, in which short bitstreams are sent instead of full-dimensional vectors of floats.<n>LoCoDL provably benefits from local training and compression and enjoys a doubly-accelerated communication complexity, with respect to the condition number of the functions and the model dimension, in the general heterogenous regime with strongly convex functions.
  arXiv  Detail & Related papers  (2024-03-07T09:22:50Z)
• Shadowheart SGD: Distributed Asynchronous SGD with Optimal Time Complexity Under Arbitrary Computation and Communication Heterogeneity [85.92481138826949]
  We develop a new method-Shadowheart SGD-that provably improves the time complexities of all previous centralized methods. We also consider the bidirectional setup, where broadcasting from the server to the workers is non-negligible, and develop a corresponding method.
  arXiv  Detail & Related papers  (2024-02-07T12:15:56Z)
• $\ extbf{A}^2\ extbf{CiD}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning [0.0]
  We introduce a principled asynchronous, randomized, gossip-based optimization algorithm which works thanks to a continuous local momentum named $textbfA2textbfCiD2$. Our theoretical analysis proves accelerated rates compared to previous asynchronous decentralized baselines. We show consistent improvement on the ImageNet dataset using up to 64 asynchronous workers.
  arXiv  Detail & Related papers  (2023-06-14T06:52:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.