SparseRL-Sync: Lossless Weight Synchronization with ~100x Less Communication

• URL: http://arxiv.org/abs/2605.07330v1
• Date: Fri, 08 May 2026 06:34:51 GMT
• Title: SparseRL-Sync: Lossless Weight Synchronization with ~100x Less Communication
• Abstract summary: We propose and implement SparseRL-Sync, which replaces full-weight transfers with a sparse update payload.<n>SparseRL-Sync significantly improves scalability and end-to-end efficiency in bandwidth-limited and highly asynchronous RL settings.
• Score: 0.6775438671102701
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In large-scale reinforcement learning (RL) systems with decoupled Trainer-Rollout execution, the Trainer must regularly synchronize policy weights to the Rollout side to limit policy staleness. When inter-node bandwidth is abundant, such synchronization is usually only a small fraction of end-to-end cost. As model size grows, however, the communication demand rises rapidly. In bandwidth-constrained or network-variable deployments -- for example, cross-datacenter or cross-cluster settings, heterogeneous resource pools, and online RL -- weight synchronization can become a dominant bottleneck for throughput and tail latency. We observe that, in mainstream large-model RL training, the locations where parameters actually change are highly sparse at the element level (often 99%+ sparsity). Building on this observation, we propose and implement SparseRL-Sync, which replaces full-weight transfers with a lossless sparse update payload (indices and values) that can be exactly reconstructed on the inference side, thereby preserving 100% fidelity. Under a simplified cost model, sparse synchronization reduces the per-update communication volume from S to approximately S/X; with 99% sparsity (X ~ 100), this yields about a 100x reduction in transmitted data. Combined with appropriate bucketing, SparseRL-Sync also reduces launch and control-plane overhead, significantly improving scalability and end-to-end efficiency in bandwidth-limited and highly asynchronous RL settings.

Related papers

• Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo [53.37011208698893]
  DiLoCo's outer synchronization remains bandwidth-heavy and brittle to stragglers and transient failures.<n>We relax exact synchronization to approximate synchronization via mixing/gossip, which degrades gracefully under delays and communication failures.<n>Our framework substantially improves compute utilization compared to DiLoCo, with training progress ranging from comparable to closely matching it, and is more robust to failures.
  arXiv  Detail & Related papers  (2026-06-22T02:15:13Z)
• Decoupled DiLoCo for Resilient Distributed Pre-training [19.201912399584813]
  We introduce Decoupled DiLoCo, an evolution of the DiLoCo framework designed to break the lock-step synchronization barrier.<n>We achieve significantly improved training efficiency in failure-prone environments with millions of simulated chips with strictly zero global downtime.
  arXiv  Detail & Related papers  (2026-04-23T08:45:38Z)
• Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL [16.40150726450328]
  In bandwidth-constrained decentralized environments, our approach achieves over 100x (14 GB to 108 MB) communication reduction.<n>We present a systematic empirical study of weight-update sparsity at both step-level and multi-step granularities.<n>We find that update sparsity is consistently high, frequently exceeding 99% across practically relevant settings.
  arXiv  Detail & Related papers  (2026-02-03T18:56:48Z)
• AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism [54.8494905524997]
  We introduce asynchronous updates across both parallelism axes, relaxing the co-location requirement.<n>We provide convergence guarantees for both sparse averaging and asynchronous updates.<n>Experiments on large-scale language models demonstrate that our approach matches the performance of the fully synchronous baseline.
  arXiv  Detail & Related papers  (2026-01-30T01:24:47Z)
• RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure [49.88201789074532]
  Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning.<n>We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure.
  arXiv  Detail & Related papers  (2025-12-27T11:14:23Z)
• Laminar: A Scalable Asynchronous RL Post-Training Framework [20.127034898123508]
  Long-tail skewness in RL trajectory generation causes severe GPU underutilization.<n>Current RL systems rely on global weight synchronization between the actor and all rollouts, which creates a rigid model update schedule.<n>We propose Laminar, a scalable and robust RL post-training system built on a fully decoupled architecture.
  arXiv  Detail & Related papers  (2025-10-14T15:29:14Z)
• Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism [59.79227116582264]
  Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging.<n>We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation.
  arXiv  Detail & Related papers  (2025-06-02T02:19:22Z)
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation [55.75008325187133]
  Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs)<n>StreamRL is designed with disaggregation from first principles to address two types of performance bottlenecks.<n> Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-04-22T14:19:06Z)
• Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch [66.84195842685459]
  Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time.<n>Recently, distributed algorithms like DiLoCo have relaxed such co-location constraint.<n>We show experimentally that we can distribute training of billion-scale parameters and reach similar quality as before.
  arXiv  Detail & Related papers  (2025-01-30T17:23:50Z)
• Accelerating Distributed ML Training via Selective Synchronization [0.0]
  textttSelSync is a practical, low-overhead method for DNN training that dynamically chooses to incur or avoid communication at each step. Our system converges to the same or better accuracy than BSP while reducing training time by up to 14$times$.
  arXiv  Detail & Related papers  (2023-07-16T05:28:59Z)
• Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning [10.196574441542646]
  Gradient Descent (SGD) has become the de facto way to train deep neural networks in distributed clusters. A critical factor in determining the training throughput and model accuracy is the choice of the parameter synchronization protocol. In this paper, we design a hybrid synchronization approach that exploits the benefits of both BSP and ASP.
  arXiv  Detail & Related papers  (2021-04-16T20:49:28Z)
• High-Throughput Synchronous Deep RL [132.43861715707905]
  We propose High-Throughput Synchronous Deep Reinforcement Learning (HTS-RL) We perform learning and rollouts concurrently, devise a system design which avoids stale policies' We evaluate our approach on Atari games and the Google Research Football environment.
  arXiv  Detail & Related papers  (2020-12-17T18:59:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.