mKernel: Fast Multi-GPU, Multi-Node Fused Kernels

• URL: http://arxiv.org/abs/2609.13585v1
• Date: Fri, 11 Sep 2026 22:50:04 GMT
• Title: mKernel: Fast Multi-GPU, Multi-Node Fused Kernels
• Abstract summary: mKernel is a library of multi-GPU, multi-node fused kernels that overlap computation, intra-node NVLink communication, and inter-node RDMA at tile granularity.<n>It achieves speedups of up to 1.72x on GEMM+AllReduce and $1.88times$ on Ring Attention.
• Score: 37.86216303017995
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granularity of kernels, on separate streams, reduces only part of this communication cost. Fused kernels often have better performance by transmitting each output tile as soon as it is produced, but existing fused kernels are largely confined to a single NVLink domain. We present mKernel, a library of multi-GPU, multi-node fused kernels that overlap computation, intra-node NVLink communication, and inter-node RDMA at tile granularity. mKernel partitions the streaming multiprocessors (SMs) of a persistent kernel into compute and communication roles, and an on-GPU controller tunes the SM partition adaptively at run time, since the best SM partition varies with the kernel and the input shape. It structures data movement hierarchically so that data traversing the inter-node network is minimized. Finally, it drives the network from the GPU through a lightweight command queue and host proxy implemented directly on RDMA verbs, which allows the same kernels to run on any network backend (e.g. InfiniBand and on AWS EFA); we observe, surprisingly, that GPUDirect Async (IBGDA) yields little additional benefit over host-assisted GPU-initiated communication. We implement five kernels spanning tensor, sequence, and expert parallelism. On two 16-GPU H200 clusters, mKernel achieves speedups of up to 1.72x on GEMM+AllReduce and $1.88\times$ on Ring Attention.

Related papers

• Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads [0.0]
  Communication overhead has become a dominant bottleneck in multi- GPU training.<n>This work explores concurrent execution of computation and collective communication using two portable runtime controls.<n>Our approach regulates computation- Kernel residency through per-block shared-memory allocation, leaving sufficient on-chip resources for communication kernels to make progress.
  arXiv  Detail & Related papers  (2026-06-08T08:33:03Z)
• Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment [60.442064966340524]
  We present a prototype study on NVIDIA Jetson Orin Nano devices connected over WiFi.<n>Key finding is that the dominant bottleneck is not just network bandwidth but also the CPU- GPU staging during communication.<n>Experiments show that this strategy reduces latency by 65%-77% and energy consumption by 34%-52% relative to full-tensor exchange.
  arXiv  Detail & Related papers  (2026-05-25T10:39:28Z)
• Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs [17.461191811780722]
  Mirage Persistent Kernel (MPK) is the first compiler and runtime system that automatically transforms multi-GPU model inference into a single high-performance megakernel.<n>MPK introduces an SM-level graph representation that captures data dependencies at the granularity of individual streaming multiprocessors.<n>MPK significantly outperforms existing kernel-per-operator serving systems by reducing end-to-end latency by up to 1.7x.
  arXiv  Detail & Related papers  (2025-12-22T14:18:20Z)
• GPU-Initiated Networking for NCCL [0.7990599798388804]
  Traditional GPU communication follows a host-initiated model, where the CPU orchestrates all communication operations.<n>This paper presents the GIN architecture, design, semantics, and highlights its impact on MoE communication.
  arXiv  Detail & Related papers  (2025-11-19T03:36:03Z)
• TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference [9.98718571193106]
  Distributed inference of large language models (LLMs) can introduce overheads of up to 20% even over GPUs connected via high-speed interconnects such as NVLink.<n>We present TokenWeave to address these challenges.<n>Our evaluations demonstrate up to 1.29x speedup in latency and 1.26x higher throughput across multiple models and workloads.
  arXiv  Detail & Related papers  (2025-05-16T14:53:50Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• Explore as a Storm, Exploit as a Raindrop: On the Benefit of Fine-Tuning Kernel Schedulers with Coordinate Descent [48.791943145735]
  We show the potential to reduce Ansor's search time while enhancing kernel quality. We apply this approach to the first 300 kernels that Ansor generates. This result has been replicated in 20 well-known deep-learning models.
  arXiv  Detail & Related papers  (2024-06-28T16:34:22Z)
• FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion [9.5114389643299]
  This paper proposes a novel method, Flux, to significantly hide communication latencies with dependent computations for GPUs. Flux can potentially overlap up to 96% of communication given a fused kernel. Overall, it can achieve up to 1.24x speedups for training over Megatron-LM on a cluster of 128 GPU with various GPU generations and interconnects.
  arXiv  Detail & Related papers  (2024-06-11T00:17:39Z)
• Spectrum-guided Multi-granularity Referring Video Object Segmentation [56.95836951559529]
  Current referring video object segmentation (R-VOS) techniques extract conditional kernels from encoded (low-resolution) vision-language features to segment the decoded high-resolution features. This causes significant feature drift, which the segmentation kernels struggle to perceive during the forward computation. We propose a Spectrum-guided Multi-granularity approach, which performs direct segmentation on the encoded features and employs visual details to further optimize the masks.
  arXiv  Detail & Related papers  (2023-07-25T14:35:25Z)
• DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks [58.48833325238537]
  Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. In this paper, we presentGNN that optimize the well-known Deep Graph Library (DGL) for full-batch training on CPU clusters. Our results on four common GNN benchmark datasets show up to 3.7x speed-up using a single CPU socket and up to 97x speed-up using 128 CPU sockets.
  arXiv  Detail & Related papers  (2021-04-14T08:46:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.