Don't Let a Few Network Failures Slow the Entire AllReduce

• URL: http://arxiv.org/abs/2606.01680v1
• Date: Mon, 01 Jun 2026 04:40:41 GMT
• Title: Don't Let a Few Network Failures Slow the Entire AllReduce
• Abstract summary: We present the first information-theoretic lower bound on AllReduce completion time under asymmetric network bandwidth.<n>We then design OptCC, a four-stage pipelined AllReduce algorithm that approaches this lower bound.<n> Experiments on SimAI confirm that OptCC closes the gap left by existing fault-tolerant schemes.
• Score: 7.373533433335005
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Network failures are among the most frequent hardware faults in large-scale GPU clusters and a leading cause of training-job interruptions. Modern collective communication libraries such as NCCL mitigate network failures by rerouting traffic through surviving NICs on the same server, trading reduced inter-node bandwidth for uninterrupted training. However, the degraded server remains on the critical path of the standard ring algorithm, slowing the entire collective. We present the first information-theoretic lower bound on AllReduce completion time under asymmetric network bandwidth and show that when the straggler retains at least half of its original bandwidth, the unavoidable overhead relative to the fault-free optimum is only O(1/p) for p GPUs. We then design OptCC, a four-stage pipelined AllReduce algorithm that approaches this lower bound. Experiments on SimAI confirm that OptCC closes the gap left by existing fault-tolerant schemes: under practical network failures with up to 50% bandwidth loss, OptCC completes AllReduce within 2-6% of NCCL's fault-free ring performance, whereas the state-of-the-art incurs up to 57% overhead.

Related papers

• Tuning Collective Patterns to Alleviate Congestion in Shared AI Clusters [1.7699712680857242]
  Distributed AI training involves recurring rounds of data exchange between multiple pairs of GPU nodes.<n>Current approaches for evading congestion in AI clusters assume global control over the entire workload.<n>We build a system, REACT, that tunes the recurring pattern of data exchange between GPU nodes in response to congestion.
  arXiv  Detail & Related papers  (2026-09-03T19:32:40Z)
• Efficient AllReduce with Stragglers [4.6987734780862604]
  StragglAR is a parallel algorithm for AllReduce that accelerates distributed training and inference.<n>StragglAR achieves a 2x theoretical speedup over popular bandwidth-efficient algorithms for large GPU clusters.<n>On an 8- GPU server, StragglAR provides a 25% speedup over state-of-the-art AllReduce algorithms.
  arXiv  Detail & Related papers  (2025-05-29T15:03:56Z)
• Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding [56.066799081747845]
  The ever-growing size of neural networks poses serious challenges on resource-constrained devices.<n>We propose a novel post-training compression framework that combines rate-aware quantization with entropy coding.<n>Our method allows for very fast decoding and is compatible with arbitrary quantization grids.
  arXiv  Detail & Related papers  (2025-05-24T15:52:49Z)
• Prioritising Interactive Flows in Data Center Networks With Central Control [0.0]
  We deal with two problems relating to central controller assisted prioritization of interactive flow in data center networks. In the first part of the thesis, we deal with the problem of congestion control in a software defined network. We propose a framework, where the controller with its global view of the network actively participates in the congestion control decisions of the end TCP hosts.
  arXiv  Detail & Related papers  (2023-10-27T07:15:15Z)
• Teal: Learning-Accelerated Optimization of WAN Traffic Engineering [68.7863363109948]
  We present Teal, a learning-based TE algorithm that leverages the parallel processing power of GPUs to accelerate TE control. To reduce the problem scale and make learning tractable, Teal employs a multi-agent reinforcement learning (RL) algorithm to independently allocate each traffic demand. Compared with other TE acceleration schemes, Teal satisfies 6--32% more traffic demand and yields 197--625x speedups.
  arXiv  Detail & Related papers  (2022-10-25T04:46:30Z)
• Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs [64.26714148634228]
  congestion control (CC) algorithms become extremely difficult to design. It is currently not possible to deploy AI models on network devices due to their limited computational capabilities. We build a computationally-light solution based on a recent reinforcement learning CC algorithm.
  arXiv  Detail & Related papers  (2022-07-05T20:42:24Z)
• 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed [17.953619054149378]
  We propose a new communication-efficient algorithm, 1-bit LAMB, which supports adaptive layerwise learning rates even when communication is compressed. For BERT-Large pre-training task with batch sizes from 8K to 64K, our evaluations demonstrate that 1-bit LAMB with NCCL-based backend is able to achieve up to 4.6x communication volume reduction.
  arXiv  Detail & Related papers  (2021-04-13T10:07:49Z)
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch [75.69506249886622]
  Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network.
  arXiv  Detail & Related papers  (2021-02-08T05:55:47Z)
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks [87.01739569518513]
  We replace conventional ReLU with Bounded ReLU and find that the decline is due to activation quantization. Our integer networks achieve equivalent performance as the corresponding FPN networks, but have only 1/4 memory cost and run 2x faster on modern GPU.
  arXiv  Detail & Related papers  (2020-06-21T08:23:03Z)
• Joint Device-Edge Inference over Wireless Links with Pruning [20.45405359815043]
  We propose a joint feature compression and transmission scheme for efficient inference at the wireless network edge. This is the first work that combines DeepJSCC with network pruning, and applies it to image classification over the wireless edge.
  arXiv  Detail & Related papers  (2020-03-04T12:06:11Z)
• Discrimination-aware Network Pruning for Deep Model Compression [79.44318503847136]
  Existing pruning methods either train from scratch with sparsity constraints or minimize the reconstruction error between the feature maps of the pre-trained models and the compressed ones. We propose a simple-yet-effective method called discrimination-aware channel pruning (DCP) to choose the channels that actually contribute to the discriminative power. Experiments on both image classification and face recognition demonstrate the effectiveness of our methods.
  arXiv  Detail & Related papers  (2020-01-04T07:07:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.