GPU-Initiated Networking for NCCL

• URL: http://arxiv.org/abs/2511.15076v2
• Date: Mon, 24 Nov 2025 23:45:45 GMT
• Title: GPU-Initiated Networking for NCCL
• Abstract summary: Traditional GPU communication follows a host-initiated model, where the CPU orchestrates all communication operations.<n>This paper presents the GIN architecture, design, semantics, and highlights its impact on MoE communication.
• Score: 0.7990599798388804
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern AI workloads, especially Mixture-of-Experts (MoE) architectures, increasingly demand low-latency, fine-grained GPU-to-GPU communication with device-side control. Traditional GPU communication follows a host-initiated model, where the CPU orchestrates all communication operations - a characteristic of the CUDA runtime. Although robust for collective operations, applications requiring tight integration of computation and communication can benefit from device-initiated communication that eliminates CPU coordination overhead. NCCL 2.28 introduces the Device API with three operation modes: Load/Store Accessible (LSA) for NVLink/PCIe, Multimem for NVLink SHARP, and GPU-Initiated Networking (GIN) for network RDMA. This paper presents the GIN architecture, design, semantics, and highlights its impact on MoE communication. GIN builds on a three-layer architecture: i) NCCL Core host-side APIs for device communicator setup and collective memory window registration; ii) Device-side APIs for remote memory operations callable from CUDA kernels; and iii) A network plugin architecture with dual semantics (GPUDirect Async Kernel-Initiated and Proxy) for broad hardware support. The GPUDirect Async Kernel-Initiated backend leverages DOCA GPUNetIO for direct GPU-to-NIC communication, while the Proxy backend provides equivalent functionality via lock-free GPU-to-CPU queues over standard RDMA networks. We demonstrate GIN's practicality through integration with DeepEP, an MoE communication library. Comprehensive benchmarking shows that GIN provides device-initiated communication within NCCL's unified runtime, combining low-latency operations with NCCL's collective algorithms and production infrastructure.

Related papers

• mKernel: Fast Multi-GPU, Multi-Node Fused Kernels [37.86216303017995]
  mKernel is a library of multi-GPU, multi-node fused kernels that overlap computation, intra-node NVLink communication, and inter-node RDMA at tile granularity.<n>It achieves speedups of up to 1.72x on GEMM+AllReduce and $1.88times$ on Ring Attention.
  arXiv  Detail & Related papers  (2026-09-11T22:50:04Z)
• Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs [27.795876706914857]
  This paper aims to enable computation- and communication-efficient GPU sharing across devices within local area networks (LANs)<n>We propose Gleam, a novel and network-efficient framework for task-generic GPU sharing.<n>Gleem consistently outperforms state-of-the-art baselines, achieving 1.4-24.2 times improvements in API remoting efficiency.
  arXiv  Detail & Related papers  (2026-07-25T09:06:47Z)
• Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment [60.442064966340524]
  We present a prototype study on NVIDIA Jetson Orin Nano devices connected over WiFi.<n>Key finding is that the dominant bottleneck is not just network bandwidth but also the CPU- GPU staging during communication.<n>Experiments show that this strategy reduces latency by 65%-77% and energy consumption by 34%-52% relative to full-tensor exchange.
  arXiv  Detail & Related papers  (2026-05-25T10:39:28Z)
• NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL [5.243473994390624]
  NCCL EP (Expert Parallelism) is a ground-up MoE communication library built entirely on NCCL's Device API.<n>It provides unified ncclEpDispatch and ncclEpCombine primitives with both C and Python interfaces.<n>We evaluate NCCL EP on an H100-based cluster across multi-node configurations, demonstrating competitive LL kernel performance and presenting end-to-end results with vLLM integration.
  arXiv  Detail & Related papers  (2026-03-13T21:28:22Z)
• CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation [51.72529978689561]
  Agent is a large-scale agentic reinforcement learning system that develops kernel expertise through three components.<n>Agent delivers 100%, 100%, and 92% faster rate over torchcompile on KernelBench.
  arXiv  Detail & Related papers  (2026-02-27T18:58:05Z)
• HetCCL: Accelerating LLM Training with Heterogeneous GPUs [5.820335672510985]
  HetCCL is a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications.<n>HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments.
  arXiv  Detail & Related papers  (2026-01-30T05:31:29Z)
• UCCL-EP: Portable Expert-Parallel Communication [34.712100988183174]
  State-of-the-art EP communication systems such as DeepEP demonstrate strong performance but exhibit poor portability across heterogeneous GPU and NIC platforms.<n>We present UCCL-EP, a portable EP communication system that delivers DeepEP-level performance across heterogeneous GPU and NIC hardware.
  arXiv  Detail & Related papers  (2025-12-22T20:05:09Z)
• FlexCTC: GPU-powered CTC Beam Decoding With Advanced Contextual Abilities [16.660841429852333]
  We present a novel open-source FlexCTC toolkit for fully-based beam decoding, designed for Connectionist Temporal Classification (CTC) models.<n>Developed entirely in Python and PyTorch, it offers a fast, user-friendly, and alternative to traditional C++, or WFST-based GPUs.<n>It also supports advanced contextualization techniques, including GPU-powered N-gram language model fusion and phrase-level boosting.
  arXiv  Detail & Related papers  (2025-08-10T12:15:57Z)
• Minute-Long Videos with Dual Parallelisms [57.22737565366549]
  Diffusion Transformer (DiT)-based video diffusion models generate high-quality videos at scale but incur prohibitive processing latency and memory costs for long videos.<n>We propose a novel distributed inference strategy, termed DualParal.<n>Instead of generating an entire video on a single GPU, we parallelize both temporal frames and model layers across GPUs.
  arXiv  Detail & Related papers  (2025-05-27T11:55:22Z)
• GPU-centric Communication Schemes for HPC and ML Applications [0.0]
  GPU-aware communication schemes move the GPU-attached communication buffers in the application directly from the GPU to the NIC without staging through the host memory.<n>A CPU thread is required to orchestrate the communication operations even with support for such GPU-awareness.<n>This survey discusses various available GPU-centric communication schemes that move the control path of the communication operations from the CPU to the GPU.
  arXiv  Detail & Related papers  (2025-03-31T15:43:18Z)
• GME: GPU-based Microarchitectural Extensions to Accelerate Homomorphic Encryption [33.87964584665433]
  Homomorphic Encryption (FHE) enables the processing of encrypted data without decrypting it. FHE introduces a slowdown of up to five orders of magnitude as compared to the same computation using plaintext data. We propose GME, which combines three key microarchitectural extensions along with a compile-time optimization to the current AMD CDNA GPU architecture.
  arXiv  Detail & Related papers  (2023-09-20T01:50:43Z)
• FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs [57.12856172329322]
  We envision a decentralized system unlocking the potential vast untapped consumer-level GPU. This system faces critical challenges, including limited CPU and GPU memory, low network bandwidth, the variability of peer and device heterogeneity.
  arXiv  Detail & Related papers  (2023-09-03T13:27:56Z)
• INR-Arch: A Dataflow Architecture and Compiler for Arbitrary-Order Gradient Computations in Implicit Neural Representation Processing [66.00729477511219]
  Given a function represented as a computation graph, traditional architectures face challenges in efficiently computing its nth-order gradient. We introduce INR-Arch, a framework that transforms the computation graph of an nth-order gradient into a hardware-optimized dataflow architecture. We present results that demonstrate 1.8-4.8x and 1.5-3.6x speedup compared to CPU and GPU baselines respectively.
  arXiv  Detail & Related papers  (2023-08-11T04:24:39Z)
• Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures [67.47328776279204]
  This work introduces a framework to develop efficient, portable Deep Learning and High Performance Computing kernels. We decompose the kernel development in two steps: 1) Expressing the computational core using Processing Primitives (TPPs) and 2) Expressing the logical loops around TPPs in a high-level, declarative fashion. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
  arXiv  Detail & Related papers  (2023-04-25T05:04:44Z)
• Synthesizing Collective Communication Algorithms for Heterogeneous Networks with TACCL [1.5528708400965123]
  We present TACCL, a synthesizer for collective communication primitives for large-scale multi-GPU systems. TACCL encodes a profiled topology and input size into a synthesis problem to generate optimized communication algorithms. Using TACCL's algorithms speeds up the end-to-end training of an internal mixture of experts model by $17%$.
  arXiv  Detail & Related papers  (2021-11-08T23:20:52Z)
• DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks [58.48833325238537]
  Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. In this paper, we presentGNN that optimize the well-known Deep Graph Library (DGL) for full-batch training on CPU clusters. Our results on four common GNN benchmark datasets show up to 3.7x speed-up using a single CPU socket and up to 97x speed-up using 128 CPU sockets.
  arXiv  Detail & Related papers  (2021-04-14T08:46:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.