RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts

• URL: http://arxiv.org/abs/2604.26039v1
• Date: Tue, 28 Apr 2026 18:20:12 GMT
• Title: RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts
• Abstract summary: We present RaMP, a routing-aware dispatch framework.<n>RaMP delivers 1.22x kernel speedup over static dispatch and 1.30x end-to-end speedup in vLLM serving over Triton, 1.41x over DeepGEMM, and 1.13x over FlashInfer CUTLASS.
• Score: 0.22625548856057365
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The optimal kernel configuration for Mixture-of-Experts (MoE) inference depends on both batch size and the expert routing distribution, yet production systems dispatch from batch size alone, leaving 10-70% of kernel throughput unrealized. We present RaMP, a routing-aware dispatch framework. A performance-region analysis derives, from hardware constants alone, when each optimization helps, correctly predicting all 8 tested architectures, including 3 unseen. A four-parameter wave cost model selects the fastest configuration from the runtime expert histogram, achieving 0.93% mean regret versus exhaustive search, fitted from just 10-24 minutes of one-time profiling per model. Because the model depends only on CTA grid geometry, it is kernel-agnostic: applied to Alpha-MoE, it delivers 1.14x with no source modification. Paired with a co-designed CuTe DSL kernel exposing 134-268 polymorphic configurations, RaMP delivers 1.22x kernel speedup over static dispatch and 1.30x end-to-end speedup in vLLM serving over Triton, 1.41x over DeepGEMM, and 1.13x over FlashInfer CUTLASS.

Related papers

• LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization [7.393546440833148]
  Large language models have become increasingly capable agents for low-level code and kernel optimization.<n>We introduce LLM4LLM, a deployment-aware closed-loop optimization framework.<n>It improves end-to-end latency for every evaluated model, achieving 3.91$times$/6.98$times$ geometric-mean speedups on A100/H100.
  arXiv  Detail & Related papers  (2026-08-22T08:12:16Z)
• PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloud [19.027069678233655]
  PhyAI is a Physical AI inference engine with a single runtime.<n>It runs vision-language-action (VLA) models and world-action models (WAMs) on single or multiple GPU.<n>It achieves 1.40x-4.65x speedups over the official implementations of pi0, pi0.5, GR00T N1.7, and MiniCPM-Robot.
  arXiv  Detail & Related papers  (2026-08-04T13:53:48Z)
• Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization [37.645358922637065]
  KernelPro is a closed-loop multi-agent system that automatically generates, profiles, and iteratively optimize kernel code.<n>On Kernel, KernelPro achieves mean speedups of 2.42x/4.69x/5.30x on Levels 1/2/3, establishing state-of-the-art performance across all difficulty levels.
  arXiv  Detail & Related papers  (2026-06-24T23:28:09Z)
• dMoE: dLLMs with Learnable Block Experts [71.572316901001]
  We propose dMoE, a simple yet effective block-level MoE framework.<n>We show that dMoE substantially reduces the number of uniquely activated experts during inference.<n>It also reduces memory usage by 76.64% to 79.84% and achieves 1.14$times$ to 1.66$times$ end-to-end latency speedup.
  arXiv  Detail & Related papers  (2026-05-29T06:03:50Z)
• Xe-Forge: Multi-Stage LLM-Powered Kernel Optimization for Intel GPU [38.81440160993858]
  Porting deep learning algorithms to new hardware accelerators requires developers to apply the same low-level optimizations to every Triton kernel in their code-base.<n>We present Xe-Forge, a multi-stage LLM-powered pipeline that automates this process for Intel GPU.<n>A curated knowledge base encodes Intel GPU constraints that are absent from LLM training data, keeping the model within architecturally valid bounds.
  arXiv  Detail & Related papers  (2026-04-16T20:21:01Z)
• COREY: Entropy-Guided Runtime Chunk Scheduling for Selective Scan Kernels [11.316541559874864]
  Prototype scheduler uses activation entropy estimated via fixed-width histograms as a runtime signal for chunk-size selection.<n>COREY is positioned as a Concept and Feasibility contribution.<n>This work does not yet include a fully integrated end-to-end run connecting Tier 2a and Tier 2b.
  arXiv  Detail & Related papers  (2026-04-12T12:07:48Z)
• MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale [36.89558970450915]
  Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware.<n>We present a methodology for designing such models using hardware-in-the-loop architecture search under mobile latency constraints.
  arXiv  Detail & Related papers  (2026-03-16T22:10:50Z)
• AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving [16.664502126572856]
  AIConfigurator is a unified performance-modeling system for Large Language Model (LLM) inference.<n>It enables rapid, framework-a configuration search without requiring GPU-based profiling.<n>It identifies superior serving configurations that improve performance by up to 40% for dense models.
  arXiv  Detail & Related papers  (2026-01-09T20:03:57Z)
• AKG kernel Agent: A Multi-Agent Framework for Cross-Platform Kernel Synthesis [13.239454996851771]
  Modern AI models demand high-performance computation kernels.<n>Akg kernel agent (AI-driven Kernel Generator) is designed to support multiple domain-specific languages.<n>System's modular design allows rapid integration of backend DSLs and hardware targets.<n>System achieves an average speedup of 1.46$times over PyTorch Eager baselines.
  arXiv  Detail & Related papers  (2025-12-29T12:42:05Z)
• SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations [54.303301888915406]
  Mixture of Experts (MoE) models have emerged as the de facto architecture for scaling up language models without significantly increasing the computational cost.<n>We propose a memory-efficient algorithm to compute the forward and backward passes of MoEs with minimal activation caching.<n>We also propose a novel "token rounding" method that minimizes the wasted compute due to padding in Grouped GEMM kernels.
  arXiv  Detail & Related papers  (2025-12-16T04:39:10Z)
• dInfer: An Efficient Inference Framework for Diffusion Language Models [54.80918957287927]
  Diffusion-based large language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) LLMs.<n>We present dInfer, an efficient and framework for dLLM inference.
  arXiv  Detail & Related papers  (2025-10-09T16:19:42Z)
• Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition [95.54406667705999]
  Pangu Embedded is an efficient Large Language Model (LLM) reasoner developed on Ascend Neural Processing Units (NPUs)<n>It addresses the significant computational costs and inference latency challenges prevalent in existing reasoning-optimized LLMs.<n>It delivers rapid responses and state-of-the-art reasoning quality within a single, unified model architecture.
  arXiv  Detail & Related papers  (2025-05-28T14:03:02Z)
• AI Accelerators for Large Language Model In-ference: Architecture Analysis and Scaling Strategies [10.520360508397237]
  Large-language models (LLMs) are driving a new wave of specialized hardware for inference.<n>This paper presents the first workload-centric, cross-architectural performance study of commercial AI accelerators.
  arXiv  Detail & Related papers  (2025-05-13T20:21:20Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models [64.34635279436054]
  Mixture-of-Experts (MoE) architectures offer a general solution to the high inference costs of large language models (LLMs) via sparse routing. We present a solution to this memory problem, in form of a new compression and execution framework called QMoE.
  arXiv  Detail & Related papers  (2023-10-25T17:24:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.