Ada-MK: Adaptive MegaKernel Optimization via Automated DAG-based Search for LLM Inference

• URL: http://arxiv.org/abs/2605.11581v1
• Date: Tue, 12 May 2026 06:04:28 GMT
• Title: Ada-MK: Adaptive MegaKernel Optimization via Automated DAG-based Search for LLM Inference
• Abstract summary: Large language models (LLMs) serve real-time inference in commercial online advertising systems.<n>MegaKernel eliminates launch overhead and interoperator round-trips by fusing multiple operators into a single persistent kernel.<n>Ada-MK: a three-dimensional shared-memory constraint model combined with K-dimension splitting that reduces peak shared memory usage by 50%.<n>On an NVIDIA L20, Ada-MK improves single-batch throughput by up to 23.6% over vanillaRT-LLM and 50.2% over vLLM.
• Score: 17.23666143382396
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When large language models (LLMs) serve real-time inference in commercial online advertising systems, end-to-end latency must be strictly bounded to the millisecond range. Yet every token generated during the decode phase triggers thousands of kernel launches, and kernel launch overhead alone can account for 14.6% of end-to-end inference time. MegaKernel eliminates launch overhead and inter-operator HBM round-trips by fusing multiple operators into a single persistent kernel. However, existing MegaKernel implementations face a fundamental tension between portability and efficiency on resource-constrained GPUs such as NVIDIA Ada: hand-tuned solutions are tightly coupled to specific architectures and lack portability, while auto-compiled approaches introduce runtime dynamic scheduling whose branch penalties are unacceptable in latency-critical settings. We observe that under a fixed deployment configuration, the optimal execution path of a MegaKernel is uniquely determined, and runtime dynamic decision-making can be entirely hoisted to compile time. Building on this insight, we propose Ada-MK: (1) a three-dimensional shared-memory constraint model combined with K-dimension splitting that reduces peak shared memory usage by 50%; (2) MLIR-based fine-grained DAG offline search that solidifies the optimal execution path, completely eliminating runtime branching; and (3) a heterogeneous hybrid inference engine that embeds MegaKernel as a plugin into TensorRT-LLM, combining high-throughput Prefill with low-latency Decode. On an NVIDIA L20, Ada-MK improves single-batch throughput by up to 23.6% over vanilla TensorRT-LLM and 50.2% over vLLM, achieving positive gains across all tested scenarios--the first industrial deployment of MegaKernel in a commercial online advertising system.

Related papers

• Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding [51.48425605758328]
  We present Faster Flash Decoding (FFD), a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding.<n>FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning.<n>It achieves up to 11.6x kernel-level speedup and scaling to 256K context length, with 2.37x end-to-end throughput improvement.
  arXiv  Detail & Related papers  (2026-08-31T15:13:20Z)
• CUDA MPC: A GPU-Native Solver for Model Predictive Control [0.0]
  Model Predictive Control (MPC) delivers constraint-aware control on systems with fast dynamics, high-dimensional models, or long horizons.<n>This paper introduces MPC, a GPU-native MPC framework that co-designs the optimization algorithm, execution model, and memory architecture for hardware.
  arXiv  Detail & Related papers  (2026-08-04T03:02:04Z)
• An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation [63.50393485685279]
  We present a vLLM-based inference pipeline for unified speech understanding and generation.<n>We overcome the shared intuition-Free Guidance (CFG) halves throughput.<n>By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput.
  arXiv  Detail & Related papers  (2026-07-02T12:55:11Z)
• KForge: LLM-Driven Cross-Platform Kernel Generation for AI Accelerators [5.703659894138959]
  Production inference increasingly targets a heterogeneous mix of accelerators.<n>This creates a systems challenge: each pipeline now requires high-performance kernels across a growing set of hardware backends and programming models.<n>We present KForge, a cross-platform framework built around an iterative refinement loop driven by two collaborating LLM-based agents.
  arXiv  Detail & Related papers  (2026-06-01T23:48:55Z)
• MATCHA: Efficient Deployment of Deep Neural Networks on Multi-Accelerator Heterogeneous Edge SoCs [45.487954219664026]
  MATCHA generates highly concurrent schedules for parallel, heterogeneous accelerators.<n>On the Tinyerf benchmark, MATCHA improves accelerator utilization and reduces latency by up to 35%.
  arXiv  Detail & Related papers  (2026-04-10T09:04:34Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• Scaling State-Space Models on Multiple GPUs with Tensor Parallelism [0.24148976266903474]
  Selective state space models (SSMs) have rapidly become a compelling backbone for large language models.<n>But in deployment, their inference performance is often bounded by the memory capacity, bandwidth, and latency limits of a single GPU.<n>This paper presents a communication-efficient TP design for selective SSM inference that addresses three practical engineering challenges.
  arXiv  Detail & Related papers  (2026-02-24T17:47:54Z)
• Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs [17.461191811780722]
  Mirage Persistent Kernel (MPK) is the first compiler and runtime system that automatically transforms multi-GPU model inference into a single high-performance megakernel.<n>MPK introduces an SM-level graph representation that captures data dependencies at the granularity of individual streaming multiprocessors.<n>MPK significantly outperforms existing kernel-per-operator serving systems by reducing end-to-end latency by up to 1.7x.
  arXiv  Detail & Related papers  (2025-12-22T14:18:20Z)
• Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMs [61.953548065938385]
  We introduce the ''Three Taxes'' (Bulk Synchronous, Inter- Kernel Data Locality, and Kernel Launch Overhead) as an analytical framework.<n>We propose moving beyond the rigid BSP model to address key inefficiencies in distributed GPU execution.<n>We observe a 10-20% speedup in end-to-end latency over BSP-based approaches.
  arXiv  Detail & Related papers  (2025-11-04T01:15:44Z)
• Hybrid Genetic Algorithm for Optimal User Order Routing: Multi-Objective Solver Optimization in CoW Protocol Batch Auctions [0.0]
  CoW Protocol batch auctions aggregate user intents and rely on to find optimal execution paths that maximize user surplus.<n>We propose a hybrid genetic algorithm for real-time solver optimization that combines a production-grade, multi-objective NSGA-II engine with adaptive instance profiling and deterministic baselines.<n>In a 14-stratum benchmark (30 seeds each), the hybrid approach yields absolute user-surplus gains of approximately 0.40-9.82 ETH on small-to-medium orders.
  arXiv  Detail & Related papers  (2025-10-24T17:05:07Z)
• xLLM Technical Report [57.13120905321185]
  We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework.<n>xLLM builds a novel decoupled service-engine architecture.<n>xLLM-Engine co-optimizes system and algorithm designs to fully saturate computing resources.
  arXiv  Detail & Related papers  (2025-10-16T13:53:47Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference [82.1584439276834]
  Transformer-based language models such as BERT provide significant accuracy improvement for a multitude of natural language processing (NLP) tasks. We present EdgeBERT, an in-depth algorithm- hardware co-design for latency-aware energy optimization for multi-task NLP.
  arXiv  Detail & Related papers  (2020-11-28T19:21:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.