Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads

• URL: http://arxiv.org/abs/2606.09200v1
• Date: Mon, 08 Jun 2026 08:33:03 GMT
• Title: Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads
• Abstract summary: Communication overhead has become a dominant bottleneck in multi- GPU training.<n>This work explores concurrent execution of computation and collective communication using two portable runtime controls.<n>Our approach regulates computation- Kernel residency through per-block shared-memory allocation, leaving sufficient on-chip resources for communication kernels to make progress.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid growth of large-scale machine learning (ML) has made distributed training across multiple GPUs a fundamental component of modern ML systems. As model sizes and computational throughput continue to increase, communication overhead has become a dominant bottleneck in multi-GPU training, particularly when computation and communication are executed sequentially. This work explores concurrent execution of computation and collective communication using two portable runtime controls: shared-memory-driven occupancy shaping for computation kernels and elevated scheduling priority for communication kernels. Our approach regulates computation-kernel residency through per-block shared-memory allocation, leaving sufficient on-chip resources for communication kernels to make progress. In addition, assigning higher priority to communication streams ensures steady communication progress once resources become available. Experiments on NVIDIA A40, A100, H100, and AMD MI250X GPUs demonstrate that the proposed method enables effective computation-communication overlap and reduces total execution time by up to 25.5 percent, without modifying vendor libraries or kernel implementations.

Related papers

• Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment [60.442064966340524]
  We present a prototype study on NVIDIA Jetson Orin Nano devices connected over WiFi.<n>Key finding is that the dominant bottleneck is not just network bandwidth but also the CPU- GPU staging during communication.<n>Experiments show that this strategy reduces latency by 65%-77% and energy consumption by 34%-52% relative to full-tensor exchange.
  arXiv  Detail & Related papers  (2026-05-25T10:39:28Z)
• RockNet: Distributed Learning on Ultra-Low-Power Devices [49.01692357536576]
  This paper presents RockNet, a new TinyML method tailored for ultra-low-power hardware.<n>By leveraging that CPS consist of multiple devices, we design a distributed learning method that integrates Machine Learning and wireless communication.<n>Our results show that a tight integration of distributed ML, distributed computing, and communication enables, for the first time, training on ultra-low-power hardware with state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2025-10-15T09:09:30Z)
• Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering [13.185314408519107]
  Generative models have achieved remarkable success across various applications, driving the demand for multi-GPU computing.<n>We propose FlashOverlap, which utilizes a novel signaling mechanism.<n>Experiments show that FlashOverlap achieves up to 1.65x speedup through overlap, outperforming existing works in most cases.
  arXiv  Detail & Related papers  (2025-04-28T06:37:57Z)
• Sparsity-Aware Communication for Distributed Graph Neural Network Training [0.41942958779358674]
  Graph Neural Networks (GNNs) are a computationally efficient method to learn embeddings and classifications on graph data.<n>GNN training has low computational intensity, making communication costs the bottleneck for scalability.<n>We develop sparsity-aware algorithms that tackle the communication bottlenecks in GNN training with three novel approaches.
  arXiv  Detail & Related papers  (2025-04-07T01:53:14Z)
• GPU-centric Communication Schemes for HPC and ML Applications [0.0]
  GPU-aware communication schemes move the GPU-attached communication buffers in the application directly from the GPU to the NIC without staging through the host memory.<n>A CPU thread is required to orchestrate the communication operations even with support for such GPU-awareness.<n>This survey discusses various available GPU-centric communication schemes that move the control path of the communication operations from the CPU to the GPU.
  arXiv  Detail & Related papers  (2025-03-31T15:43:18Z)
• Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning [2.685330831042324]
  We propose a collection of communication and optimization strategies for ZeRO++ to reduce communication costs and improve memory utilization.<n>For a 20B GPT model, we observe a 1.71x increase in TFLOPS per GPU when compared with ZeRO++ up to 384 GCDs and a scaling efficiency of 0.94 for up to 384 GCDs.
  arXiv  Detail & Related papers  (2025-01-08T04:19:57Z)
• FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion [9.5114389643299]
  This paper proposes a novel method, Flux, to significantly hide communication latencies with dependent computations for GPUs. Flux can potentially overlap up to 96% of communication given a fused kernel. Overall, it can achieve up to 1.24x speedups for training over Megatron-LM on a cluster of 128 GPU with various GPU generations and interconnects.
  arXiv  Detail & Related papers  (2024-06-11T00:17:39Z)
• FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs [57.12856172329322]
  We envision a decentralized system unlocking the potential vast untapped consumer-level GPU. This system faces critical challenges, including limited CPU and GPU memory, low network bandwidth, the variability of peer and device heterogeneity.
  arXiv  Detail & Related papers  (2023-09-03T13:27:56Z)
• Asynchronous Parallel Incremental Block-Coordinate Descent for Decentralized Machine Learning [55.198301429316125]
  Machine learning (ML) is a key technique for big-data-driven modelling and analysis of massive Internet of Things (IoT) based intelligent and ubiquitous computing. For fast-increasing applications and data amounts, distributed learning is a promising emerging paradigm since it is often impractical or inefficient to share/aggregate data. This paper studies the problem of training an ML model over decentralized systems, where data are distributed over many user devices.
  arXiv  Detail & Related papers  (2022-02-07T15:04:15Z)
• Collaborative Learning over Wireless Networks: An Introductory Overview [84.09366153693361]
  We will mainly focus on collaborative training across wireless devices. Many distributed optimization algorithms have been developed over the last decades. They provide data locality; that is, a joint model can be trained collaboratively while the data available at each participating device remains local.
  arXiv  Detail & Related papers  (2021-12-07T20:15:39Z)
• DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks [58.48833325238537]
  Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. In this paper, we presentGNN that optimize the well-known Deep Graph Library (DGL) for full-batch training on CPU clusters. Our results on four common GNN benchmark datasets show up to 3.7x speed-up using a single CPU socket and up to 97x speed-up using 128 CPU sockets.
  arXiv  Detail & Related papers  (2021-04-14T08:46:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.