Tuning Collective Patterns to Alleviate Congestion in Shared AI Clusters

• URL: http://arxiv.org/abs/2609.04417v1
• Date: Thu, 03 Sep 2026 19:32:40 GMT
• Title: Tuning Collective Patterns to Alleviate Congestion in Shared AI Clusters
• Abstract summary: Distributed AI training involves recurring rounds of data exchange between multiple pairs of GPU nodes.<n>Current approaches for evading congestion in AI clusters assume global control over the entire workload.<n>We build a system, REACT, that tunes the recurring pattern of data exchange between GPU nodes in response to congestion.
• Score: 1.7699712680857242
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Distributed AI training involves recurring rounds of data exchange between multiple pairs of GPU nodes. Slowdown in even one flow due to congestion can cause the entire communication round to slowdown. Current approaches for evading congestion in AI clusters assume global control over the entire workload (e.g. coordinating the schedule of all jobs) or assume infrastructural support (e.g. adaptive routing in switches). They are thus ill-suited in a shared cloud setting where AI jobs belonging to one user can face external congestion from other users' jobs or background traffic beyond its own control. In this paper, we build a system, REACT, that tunes the recurring pattern of data exchange between GPU nodes (known as communication collectives) in response to congestion. REACT works at the application (communication library) layer, where it detects congestion at runtime using readily available flow stats, and tunes the collective pattern to alleviate congestion - changing the set of incident flows while retaining the semantics of information exchange (e.g. selecting which node aggregates data in an AllReduce tree). REACT requires no explicit support from the underlying network infrastructure and can be unilaterally deployed by individual users in a shared cloud setting. We prototype REACT as a shim layer over NCCL, and evaluate it on a shared academic GPU cluster - enabling REACT improves communication performance (algorithm bandwidth) by 13%-38% under network congestion. Our simulations across a range of congestion scenarios further reveal up to 75% performance improvement, highlighting the effectiveness of our approach.

Related papers

• Collective Communication Profiling of Modern-day Machine Learning Workloads [2.2816873176308907]
  Machine Learning jobs involve periodic communication using operations like AllReduce, AllGather, and Broadcast.<n>These operations may create high bandwidth and bursty traffic patterns, leading to network congestion and packet loss.<n>It is imperative to analyze these patterns, which can be helpful in provisioning network resources depending on the type of machine learning workloads.
  arXiv  Detail & Related papers  (2025-07-03T20:59:36Z)
• Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch [66.84195842685459]
  Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time.<n>Recently, distributed algorithms like DiLoCo have relaxed such co-location constraint.<n>We show experimentally that we can distribute training of billion-scale parameters and reach similar quality as before.
  arXiv  Detail & Related papers  (2025-01-30T17:23:50Z)
• FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression [55.992528247880685]
  Decentralized training faces significant challenges regarding system design and efficiency. We present FusionLLM, a decentralized training system designed and implemented for training large deep neural networks (DNNs) We show that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.
  arXiv  Detail & Related papers  (2024-10-16T16:13:19Z)
• Over-the-Air Federated Edge Learning with Hierarchical Clustering [21.51594138166343]
  In over-the-air (OTA) aggregation, mobile users (MUs) aim to reach a consensus on a global model with the help of a parameter server (PS) In OTA FL, MUs train their models using local data at every training round and transmit their gradients simultaneously using the same frequency band in an uncoded fashion. While the OTA FL has a significantly decreased communication cost, it is susceptible to adverse channel effects and noise. We propose a wireless-based hierarchical FL scheme that uses intermediate servers (ISs) to form clusters at the areas where the MUs are more densely located
  arXiv  Detail & Related papers  (2022-07-19T12:42:12Z)
• Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs [64.26714148634228]
  congestion control (CC) algorithms become extremely difficult to design. It is currently not possible to deploy AI models on network devices due to their limited computational capabilities. We build a computationally-light solution based on a recent reinforcement learning CC algorithm.
  arXiv  Detail & Related papers  (2022-07-05T20:42:24Z)
• Improving the Performance of a NoC-based CNN Accelerator with Gather Support [6.824747267214373]
  Deep learning technology drives the need for an efficient parallel computing architecture for CNNs. The CNN workload introduces many-to-one traffic in addition to one-to-one and one-to-many traffic. We propose to use the gather packet on mesh-based NoCs employing output stationary systolic array in support of many-to-one traffic.
  arXiv  Detail & Related papers  (2021-08-01T23:33:40Z)
• Cloud Collectives: Towards Cloud-aware Collectives forML Workloads with Rank Reordering [8.81194405760133]
  Cloud Collectives is a prototype that accelerates collectives by reorderingranks of participating frameworks. Collectives is non-intrusive, requires no code changes nor rebuild of an existing application, and runs without support from cloud providers. Preliminary application of Cloud Collectives on allreduce operations in public clouds results in a speedup of up to 3.7x in multiple microbenchmarks and 1.3x in real-world workloads.
  arXiv  Detail & Related papers  (2021-05-28T20:14:38Z)
• Gradient Coding with Dynamic Clustering for Straggler-Tolerant Distributed Learning [55.052517095437]
  gradient descent (GD) is widely employed to parallelize the learning task by distributing the dataset across multiple workers. A significant performance bottleneck for the per-iteration completion time in distributed synchronous GD is $straggling$ workers. Coded distributed techniques have been introduced recently to mitigate stragglers and to speed up GD iterations by assigning redundant computations to workers. We propose a novel dynamic GC scheme, which assigns redundant data to workers to acquire the flexibility to choose from among a set of possible codes depending on the past straggling behavior.
  arXiv  Detail & Related papers  (2021-03-01T18:51:29Z)
• Joint Parameter-and-Bandwidth Allocation for Improving the Efficiency of Partitioned Edge Learning [73.82875010696849]
  Machine learning algorithms are deployed at the network edge for training artificial intelligence (AI) models. This paper focuses on the novel joint design of parameter (computation load) allocation and bandwidth allocation.
  arXiv  Detail & Related papers  (2020-03-10T05:52:15Z)
• Reinforcement Learning Based Vehicle-cell Association Algorithm for Highly Mobile Millimeter Wave Communication [53.47785498477648]
  This paper investigates the problem of vehicle-cell association in millimeter wave (mmWave) communication networks. We first formulate the user state (VU) problem as a discrete non-vehicle association optimization problem. The proposed solution achieves up to 15% gains in terms sum of user complexity and 20% reduction in VUE compared to several baseline designs.
  arXiv  Detail & Related papers  (2020-01-22T08:51:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.