The Big Send-off: High Performance Collectives on GPU-based Supercomputers

• URL: http://arxiv.org/abs/2504.18658v1
• Date: Fri, 25 Apr 2025 19:23:46 GMT
• Title: The Big Send-off: High Performance Collectives on GPU-based Supercomputers
• Abstract summary: Existing libraries such as RCCL and Cray-MPICH exhibit critical limitations on systems such as Frontier.<n>We introduce PCCL, a communication library with highly optimized implementations of all-gather and reduce-scatter operations.<n>It achieves substantial performance improvements, delivering 6-33x speedups over RCCL and 28-70x over Cray-MPICH for all-gather on 2048 GCDs of Frontier.
• Score: 1.852182371724987
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We evaluate the current state of collective communication on GPU-based supercomputers for large language model (LLM) training at scale. Existing libraries such as RCCL and Cray-MPICH exhibit critical limitations on systems such as Frontier -- Cray-MPICH underutilizes network and compute resources, while RCCL suffers from severe scalability issues. To address these challenges, we introduce PCCL, a communication library with highly optimized implementations of all-gather and reduce-scatter operations tailored for distributed deep learning workloads. PCCL is designed to maximally utilize all available network and compute resources and to scale efficiently to thousands of GPUs. It achieves substantial performance improvements, delivering 6-33x speedups over RCCL and 28-70x over Cray-MPICH for all-gather on 2048 GCDs of Frontier. These gains translate directly to end-to-end performance: in large-scale GPT-3-style training, PCCL provides up to 60% and 40% speedups over RCCL for 7B and 13B parameter models, respectively.

Related papers

• CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems [2.870762512009438]
  We introduce CTA-pipelining, an execution paradigm designed to exploit shared-memory multi-GPU systems.<n>As a latency-oriented spatial scaling technique, CTA-pipelining leverages the Cooperative Thread Array level.<n>Results show CTA-pipelining reduces latency by up to 31.8% compared to micro-batching.
  arXiv  Detail & Related papers  (2026-07-08T18:54:08Z)
• Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models [67.17268530365189]
  This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models.<n>It combines and specializes various existing and novel parallelism techniques at different layers and stages of the MoE model training pipeline.
  arXiv  Detail & Related papers  (2026-07-02T08:06:57Z)
• ImageHD: Energy-Efficient On-Device Continual Learning of Visual Representations via Hyperdimensional Computing [0.07646713951724012]
  ImageHD is an FPGA accelerator for on-device continual learning of visual data based on HDC.<n>On CORe50, ImageHD achieves up to 40.4x (4.84x) speedup and 383x (105.1x) energy efficiency over optimized CPU ( GPU) baselines.
  arXiv  Detail & Related papers  (2026-04-23T04:55:18Z)
• HetCCL: Accelerating LLM Training with Heterogeneous GPUs [5.820335672510985]
  HetCCL is a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications.<n>HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments.
  arXiv  Detail & Related papers  (2026-01-30T05:31:29Z)
• RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure [49.88201789074532]
  Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning.<n>We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure.
  arXiv  Detail & Related papers  (2025-12-27T11:14:23Z)
• LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs [45.51664355320938]
  We present an end-to-end/C++ implementation for medium-sized language-model training, e.g. 3B to 32B parameters, on commodity GPUs.<n>This is achieved while executing a standard 8-bit training pipeline, without additional algorithmic approximations, and maintaining FLOP utilization of around 50%.
  arXiv  Detail & Related papers  (2025-12-17T10:51:45Z)
• Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMs [61.953548065938385]
  We introduce the ''Three Taxes'' (Bulk Synchronous, Inter- Kernel Data Locality, and Kernel Launch Overhead) as an analytical framework.<n>We propose moving beyond the rigid BSP model to address key inefficiencies in distributed GPU execution.<n>We observe a 10-20% speedup in end-to-end latency over BSP-based approaches.
  arXiv  Detail & Related papers  (2025-11-04T01:15:44Z)
• Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning [2.685330831042324]
  We propose a collection of communication and optimization strategies for ZeRO++ to reduce communication costs and improve memory utilization.<n>For a 20B GPT model, we observe a 1.71x increase in TFLOPS per GPU when compared with ZeRO++ up to 384 GCDs and a scaling efficiency of 0.94 for up to 384 GCDs.
  arXiv  Detail & Related papers  (2025-01-08T04:19:57Z)
• FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resources [45.40926501138365]
  We introduce FastCLIP, a general CLIP training framework built on advanced compositional optimization techniques. Our framework is equipped with an efficient gradient reduction strategy to reduce communication overhead. We benchmark the performance of FastCLIP and the state-of-the-art training baseline on different compute scales.
  arXiv  Detail & Related papers  (2024-07-01T16:37:18Z)
• FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs [57.12856172329322]
  We envision a decentralized system unlocking the potential vast untapped consumer-level GPU. This system faces critical challenges, including limited CPU and GPU memory, low network bandwidth, the variability of peer and device heterogeneity.
  arXiv  Detail & Related papers  (2023-09-03T13:27:56Z)
• Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection [59.77647907277523]
  Adversarial contrast learning (ACL) does not require expensive data annotations but outputs a robust representation that withstands adversarial attacks. ACL needs tremendous running time to generate the adversarial variants of all training data. This paper proposes a robustness-aware coreset selection (RCS) method to speed up ACL.
  arXiv  Detail & Related papers  (2023-02-08T03:20:14Z)
• Implementing Reinforcement Learning Datacenter Congestion Control in NVIDIA NICs [64.26714148634228]
  congestion control (CC) algorithms become extremely difficult to design. It is currently not possible to deploy AI models on network devices due to their limited computational capabilities. We build a computationally-light solution based on a recent reinforcement learning CC algorithm.
  arXiv  Detail & Related papers  (2022-07-05T20:42:24Z)
• Synthesizing Collective Communication Algorithms for Heterogeneous Networks with TACCL [1.5528708400965123]
  We present TACCL, a synthesizer for collective communication primitives for large-scale multi-GPU systems. TACCL encodes a profiled topology and input size into a synthesis problem to generate optimized communication algorithms. Using TACCL's algorithms speeds up the end-to-end training of an internal mixture of experts model by $17%$.
  arXiv  Detail & Related papers  (2021-11-08T23:20:52Z)
• High Performance Hyperspectral Image Classification using Graphics Processing Units [0.0]
  Real-time remote sensing applications require onboard real time processing capabilities. Lightweight, small size and low power consumption hardware is essential for onboard real time processing systems.
  arXiv  Detail & Related papers  (2021-05-30T09:26:03Z)
• DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks [58.48833325238537]
  Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. In this paper, we presentGNN that optimize the well-known Deep Graph Library (DGL) for full-batch training on CPU clusters. Our results on four common GNN benchmark datasets show up to 3.7x speed-up using a single CPU socket and up to 97x speed-up using 128 CPU sockets.
  arXiv  Detail & Related papers  (2021-04-14T08:46:35Z)
• Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architectures [56.69373580921888]
  We focus on Recommender Systems which account for most of the AI cycles in cloud computing centers. By enabling it to run on latest CPU hardware and software tailored for HPC, we are able to achieve more than two-orders of magnitude improvement in performance.
  arXiv  Detail & Related papers  (2020-05-10T14:40:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.