NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL

• URL: http://arxiv.org/abs/2603.13606v1
• Date: Fri, 13 Mar 2026 21:28:22 GMT
• Title: NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL
• Abstract summary: NCCL EP (Expert Parallelism) is a ground-up MoE communication library built entirely on NCCL's Device API.<n>It provides unified ncclEpDispatch and ncclEpCombine primitives with both C and Python interfaces.<n>We evaluate NCCL EP on an H100-based cluster across multi-node configurations, demonstrating competitive LL kernel performance and presenting end-to-end results with vLLM integration.
• Score: 5.243473994390624
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, driving the development of specialized device-initiated communication libraries such as DeepEP, Hybrid-EP, and others. These libraries demonstrate the performance benefits of GPU-initiated RDMA for MoE dispatch and combine operations. This paper presents NCCL EP (Expert Parallelism), a ground-up MoE communication library built entirely on NCCL's Device API. NCCL EP provides unified ncclEpDispatch and ncclEpCombine primitives with both C and Python interfaces, supporting Low-Latency (LL) mode for inference decoding and High-Throughput (HT) mode for training and inference prefill. LL targets small batch sizes (1-128 tokens) using direct all-to-all RDMA+NVLink mesh connectivity with double-buffered communication for overlapping dispatch and combine phases. HT targets large batches (4096+ tokens) using hierarchical communication that aggregates tokens within NVLink domains before inter-node RDMA transmission. Both modes leverage Device API for both intra- and inter-node communications, taking advantage of its topology awareness and optimized GPU-initiated implementation. We evaluate NCCL EP on an H100-based cluster across multi-node configurations, demonstrating competitive LL kernel performance and presenting end-to-end results with vLLM integration. By building MoE communication natively within NCCL, NCCL EP provides a supported path for expert parallelism on current and emerging NVIDIA platforms.

Related papers

• mKernel: Fast Multi-GPU, Multi-Node Fused Kernels [37.86216303017995]
  mKernel is a library of multi-GPU, multi-node fused kernels that overlap computation, intra-node NVLink communication, and inter-node RDMA at tile granularity.<n>It achieves speedups of up to 1.72x on GEMM+AllReduce and $1.88times$ on Ring Attention.
  arXiv  Detail & Related papers  (2026-09-11T22:50:04Z)
• An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation [63.50393485685279]
  We present a vLLM-based inference pipeline for unified speech understanding and generation.<n>We overcome the shared intuition-Free Guidance (CFG) halves throughput.<n>By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput.
  arXiv  Detail & Related papers  (2026-07-02T12:55:11Z)
• Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment [60.442064966340524]
  We present a prototype study on NVIDIA Jetson Orin Nano devices connected over WiFi.<n>Key finding is that the dominant bottleneck is not just network bandwidth but also the CPU- GPU staging during communication.<n>Experiments show that this strategy reduces latency by 65%-77% and energy consumption by 34%-52% relative to full-tensor exchange.
  arXiv  Detail & Related papers  (2026-05-25T10:39:28Z)
• HetCCL: Accelerating LLM Training with Heterogeneous GPUs [5.820335672510985]
  HetCCL is a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications.<n>HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments.
  arXiv  Detail & Related papers  (2026-01-30T05:31:29Z)
• UCCL-EP: Portable Expert-Parallel Communication [34.712100988183174]
  State-of-the-art EP communication systems such as DeepEP demonstrate strong performance but exhibit poor portability across heterogeneous GPU and NIC platforms.<n>We present UCCL-EP, a portable EP communication system that delivers DeepEP-level performance across heterogeneous GPU and NIC hardware.
  arXiv  Detail & Related papers  (2025-12-22T20:05:09Z)
• GPU-Initiated Networking for NCCL [0.7990599798388804]
  Traditional GPU communication follows a host-initiated model, where the CPU orchestrates all communication operations.<n>This paper presents the GIN architecture, design, semantics, and highlights its impact on MoE communication.
  arXiv  Detail & Related papers  (2025-11-19T03:36:03Z)
• CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks [57.95170323315603]
  We introduce CollaPipe, a distributed learning framework that integrates collaborative pipeline parallelism with federated aggregation to support self-evolving networks.<n>In CollaPipe, the encoder part is adaptively partitioned into variable-sized segments and deployed across mobile devices for pipeline-parallel training, while the decoder is deployed on edge servers to handle generative tasks.<n>To enhance training efficiency, we formulate a joint optimization problem that adaptively allocates model segments, micro-batches, bandwidth, and transmission power.
  arXiv  Detail & Related papers  (2025-09-24T07:54:01Z)
• ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive [38.22906887556149]
  Large language model (LLM) decoding suffers from high latency due to fragmented execution across operators.<n>We introduce two cluster-level communication primitives, ClusterGather and ClusterFusion.<n>We design ClusterFusion, an execution framework that schedules communication and jointly to expand operator fusion scope.
  arXiv  Detail & Related papers  (2025-08-26T09:29:23Z)
• EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference [49.94169109038806]
  This paper introduces EPS-MoE, a novel expert pipeline scheduler for MoE that surpasses the existing parallelism schemes.<n>Our results demonstrate at most 52.4% improvement in prefill throughput compared to existing parallel inference methods.
  arXiv  Detail & Related papers  (2024-10-16T05:17:49Z)
• Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures [67.47328776279204]
  This work introduces a framework to develop efficient, portable Deep Learning and High Performance Computing kernels. We decompose the kernel development in two steps: 1) Expressing the computational core using Processing Primitives (TPPs) and 2) Expressing the logical loops around TPPs in a high-level, declarative fashion. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
  arXiv  Detail & Related papers  (2023-04-25T05:04:44Z)
• An Adaptive Device-Edge Co-Inference Framework Based on Soft Actor-Critic [72.35307086274912]
  High-dimension parameter model and large-scale mathematical calculation restrict execution efficiency, especially for Internet of Things (IoT) devices. We propose a new Deep Reinforcement Learning (DRL)-Soft Actor Critic for discrete (SAC-d), which generates the emphexit point, emphexit point, and emphcompressing bits by soft policy iterations. Based on the latency and accuracy aware reward design, such an computation can well adapt to the complex environment like dynamic wireless channel and arbitrary processing, and is capable of supporting the 5G URL
  arXiv  Detail & Related papers  (2022-01-09T09:31:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.