Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs

• URL: http://arxiv.org/abs/2512.22219v1
• Date: Mon, 22 Dec 2025 14:18:20 GMT
• Title: Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs
• Authors: Xinhao Cheng, Zhihao Zhang, Yu Zhou, Jianan Ji, Jinchen Jiang, Zepeng Zhao, Ziruo Xiao, Zihao Ye, Yingyi Huang, Ruihang Lai, Hongyi Jin, Bohan Hou, Mengdi Wu, Yixin Dong, Anthony Yip, Zihao Ye, Songting Wang, Wenqin Yang, Xupeng Miao, Tianqi Chen, Zhihao Jia,
• Abstract summary: Mirage Persistent Kernel (MPK) is the first compiler and runtime system that automatically transforms multi-GPU model inference into a single high-performance megakernel.<n>MPK introduces an SM-level graph representation that captures data dependencies at the granularity of individual streaming multiprocessors.<n>MPK significantly outperforms existing kernel-per-operator serving systems by reducing end-to-end latency by up to 1.7x.
• Score: 17.461191811780722
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Mirage Persistent Kernel (MPK), the first compiler and runtime system that automatically transforms multi-GPU model inference into a single high-performance megakernel. MPK introduces an SM-level graph representation that captures data dependencies at the granularity of individual streaming multiprocessors (SMs), enabling cross-operator software pipelining, fine-grained kernel overlap, and other previously infeasible GPU optimizations. The MPK compiler lowers tensor programs into highly optimized SM-level task graphs and generates optimized CUDA implementations for all tasks, while the MPK in-kernel parallel runtime executes these tasks within a single mega-kernel using decentralized scheduling across SMs. Together, these components provide end-to-end kernel fusion with minimal developer effort, while preserving the flexibility of existing programming models. Our evaluation shows that MPK significantly outperforms existing kernel-per-operator LLM serving systems by reducing end-to-end inference latency by up to 1.7x, pushing LLM inference performance close to hardware limits. MPK is publicly available at https://github.com/mirage-project/mirage.

Related papers

• A Tensor Compiler for Processing-In-Memory Architectures [8.353569627672622]
  Processing-In-Memory (PIM) devices can accelerate memory-intensive kernels in Machine Learning (ML) models, including Large Language Models (LLMs)<n>Current compilation approaches lack systematic optimization for diverse ML kernels across multiple PIM backends.<n>We design DCC, the first data-centric ML compiler for PIM systems that jointly co-optimizes data rearrangements and compute code optimization.
  arXiv  Detail & Related papers  (2025-11-19T14:58:16Z)
• Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMs [61.953548065938385]
  We introduce the ''Three Taxes'' (Bulk Synchronous, Inter- Kernel Data Locality, and Kernel Launch Overhead) as an analytical framework.<n>We propose moving beyond the rigid BSP model to address key inefficiencies in distributed GPU execution.<n>We observe a 10-20% speedup in end-to-end latency over BSP-based approaches.
  arXiv  Detail & Related papers  (2025-11-04T01:15:44Z)
• xLLM Technical Report [57.13120905321185]
  We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework.<n>xLLM builds a novel decoupled service-engine architecture.<n>xLLM-Engine co-optimizes system and algorithm designs to fully saturate computing resources.
  arXiv  Detail & Related papers  (2025-10-16T13:53:47Z)
• CUDA-LLM: LLMs Can Write Efficient CUDA Kernels [9.287036563375617]
  Large Language Models (LLMs) have demonstrated strong capabilities in general-purpose code generation.<n>We propose a novel framework called textbfFeature SearchReinforcement (FSR) FSR jointly optimize compilation and functional correctness.
  arXiv  Detail & Related papers  (2025-06-10T10:51:03Z)
• PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System [13.678531084541666]
  We propose PAPI, a PIM-enabled heterogeneous architecture that exploits dynamic scheduling of compute-bound or memory-bound kernels to suitable hardware units.<n>PAPI achieves 1.8$times$ and 11.1$times$ speed over a state-of-the-art heterogeneous accelerator and a state-of-the-art PIM-only accelerator.
  arXiv  Detail & Related papers  (2025-02-21T13:52:31Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• Spectrum-guided Multi-granularity Referring Video Object Segmentation [56.95836951559529]
  Current referring video object segmentation (R-VOS) techniques extract conditional kernels from encoded (low-resolution) vision-language features to segment the decoded high-resolution features. This causes significant feature drift, which the segmentation kernels struggle to perceive during the forward computation. We propose a Spectrum-guided Multi-granularity approach, which performs direct segmentation on the encoded features and employs visual details to further optimize the masks.
  arXiv  Detail & Related papers  (2023-07-25T14:35:25Z)
• Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures [67.47328776279204]
  This work introduces a framework to develop efficient, portable Deep Learning and High Performance Computing kernels. We decompose the kernel development in two steps: 1) Expressing the computational core using Processing Primitives (TPPs) and 2) Expressing the logical loops around TPPs in a high-level, declarative fashion. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
  arXiv  Detail & Related papers  (2023-04-25T05:04:44Z)
• Local Sample-weighted Multiple Kernel Clustering with Consensus Discriminative Graph [73.68184322526338]
  Multiple kernel clustering (MKC) is committed to achieving optimal information fusion from a set of base kernels. This paper proposes a novel local sample-weighted multiple kernel clustering model. Experimental results demonstrate that our LSWMKC possesses better local manifold representation and outperforms existing kernel or graph-based clustering algo-rithms.
  arXiv  Detail & Related papers  (2022-07-05T05:00:38Z)
• SMASH: Sparse Matrix Atomic Scratchpad Hashing [0.0]
  In this thesis, we introduce a novel SpGEMM kernel implementation based on the row-wise product approach. We leverage atomic instructions to merge intermediate partial products as they are generated. Our kernel can achieve 9.4x speedup as compared to competing approaches.
  arXiv  Detail & Related papers  (2021-05-29T00:22:50Z)
• FusedMM: A Unified SDDMM-SpMM Kernel for Graph Embedding and Graph Neural Networks [3.577310844634503]
  We develop a fused matrix multiplication kernel that unifies sampled dense-dense matrix multiplication and sparse-dense matrix multiplication under a single operation called FusedMM. By using user-defined functions, FusedMM can capture almost all computational patterns needed by popular graph embedding and GNN approaches.
  arXiv  Detail & Related papers  (2020-11-07T18:06:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.