Making LLMs Optimize Multi-Scenario CUDA Kernels Like Experts

• URL: http://arxiv.org/abs/2603.07169v1
• Date: Sat, 07 Mar 2026 12:22:43 GMT
• Title: Making LLMs Optimize Multi-Scenario CUDA Kernels Like Experts
• Abstract summary: Current automated kernel optimization methods focus on machine learning applications, while overlooking broader domains like sparse matrix operations in scientific computing.<n>In this paper, we address the absence of systematic evaluation for multi-scenario settings introducing MSBench.<n>Building on this benchmark, we introduce MSMaster, a hardware-aware system for kernel optimization that leverages GPU profiling information and automatically constructs the full compilation and execution toolchain.
• Score: 25.264078143000045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optimizing GPU kernels manually is a challenging and time-consuming task. With the rapid development of LLMs, automated GPU kernel optimization is gradually becoming a tangible reality. However, current LLM-driven automated optimization methods narrowly focus on machine learning applications, such as PyTorch operator optimization, while overlooking broader domains like sparse matrix operations in scientific computing. Extending to these broader applications brings new challenges for the benchmark and algorithm. Therefore, developing a general-purpose automated kernel optimization method becomes our primary focus. In this paper, we address the absence of systematic evaluation for multi-scenario settings by introducing MSKernelBench, which spans multiple scenarios, including fundamental algebraic operations, common LLM kernels, sparse matrix operators, and scientific computing routines, each supporting both FP32 and BF16 precision. Building on this benchmark, we introduce CUDAMaster, a multi-agent, hardware-aware system for kernel optimization that leverages profiling information and automatically constructs the full compilation and execution toolchain. Experimental results demonstrate that CUDAMaster achieves significant speedups across most operators, outperforming Astra by about 35%. In several cases, its performance matches or surpasses that of highly optimized, closed-source libraries such as cuBLAS. A demo showcasing the original and optimized code for each operator is available at https://hanyx2021.github.io/MSKernelBenchDemo/.

Related papers

• HIERA: Workload-Aware Planning Across Implementation Spaces for GPU Kernel Optimization [17.600449573236194]
  We propose textscHIERA, a hierarchical search-space planning framework for kernel optimization.<n>textscHIERA constructs contract-augmented task specifications, selects an appropriate implementation space across PyTorch operators, libraries, and custom kernels.<n>Experiments show textscHIERA delivers stronger overall implementation refinement, validity, and optimization performance than existing training-free methods.
  arXiv  Detail & Related papers  (2026-08-21T14:29:20Z)
• GPU Forecasters: Language Models as Selective Surrogates for Kernel Runtime Optimization [73.92934811090163]
  We study how LLMs can serve as selective GPU surrogates for kernel evaluation.<n>We measure whether their forecasts are accurate, calibrated, and practically useful for recovering fast kernels under limited GPU-measurement budgets.<n>Our experiments demonstrate that LLMs can accurately forecast relative kernel performance, that their utility can be improved through reinforcement learning.
  arXiv  Detail & Related papers  (2026-05-29T15:56:08Z)
• CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation [51.72529978689561]
  Agent is a large-scale agentic reinforcement learning system that develops kernel expertise through three components.<n>Agent delivers 100%, 100%, and 92% faster rate over torchcompile on KernelBench.
  arXiv  Detail & Related papers  (2026-02-27T18:58:05Z)
• KernelBlaster: Continual Cross-Task CUDA Optimization via Memory-Augmented In-Context Reinforcement Learning [3.4998382481249286]
  We release KernelBlaster as an open-source agentic framework, accompanied by a test harness, verification components, and a reproducible evaluation.<n>Our method achieves mean speed-ups of 1.43x, 2.50x, and 1.50x on KernelBench Levels 1, 2, and 3, respectively.
  arXiv  Detail & Related papers  (2026-02-15T19:48:43Z)
• Optimizing PyTorch Inference with LLM-Based Multi-Agent Systems [1.2289544895833646]
  We present a framework for comparing multi-agent PyTorch optimization systems.<n>We show that exploit-heavy strategies perform best when paired with error-fixing agents.<n>The best implementation achieves an average 2.88x speedup on an H100 GPU.
  arXiv  Detail & Related papers  (2025-11-21T05:37:38Z)
• STARK: Strategic Team of Agents for Refining Kernels [23.717055490630596]
  We introduce an agentic framework for GPU kernel optimization that explores the design space through multi-agent collaboration.<n>This framework mimics the workflow of expert engineers, enabling LLMs to reason about hardware trade-offs, incorporate profiling feedback, and refine kernels iteratively.<n>We evaluate our approach on KernelBench, a benchmark for LLM-based kernel optimization, and demonstrate substantial improvements over baseline agents.
  arXiv  Detail & Related papers  (2025-10-19T20:41:46Z)
• Astra: A Multi-Agent System for GPU Kernel Performance Optimization [10.715861478214961]
  We introduce Astra, the first multi-agent system for GPU kernel optimization.<n>Within Astra, specialized agents collaborate through code generation, profiling, and planning to produce kernels that are both correct and high-performance.
  arXiv  Detail & Related papers  (2025-09-09T08:39:50Z)
• CUDA-LLM: LLMs Can Write Efficient CUDA Kernels [9.287036563375617]
  Large Language Models (LLMs) have demonstrated strong capabilities in general-purpose code generation.<n>We propose a novel framework called textbfFeature SearchReinforcement (FSR) FSR jointly optimize compilation and functional correctness.
  arXiv  Detail & Related papers  (2025-06-10T10:51:03Z)
• Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures [67.47328776279204]
  This work introduces a framework to develop efficient, portable Deep Learning and High Performance Computing kernels. We decompose the kernel development in two steps: 1) Expressing the computational core using Processing Primitives (TPPs) and 2) Expressing the logical loops around TPPs in a high-level, declarative fashion. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
  arXiv  Detail & Related papers  (2023-04-25T05:04:44Z)
• Learning to Superoptimize Real-world Programs [79.4140991035247]
  We propose a framework to learn to superoptimize real-world programs by using neural sequence-to-sequence models. We introduce the Big Assembly benchmark, a dataset consisting of over 25K real-world functions mined from open-source projects in x86-64 assembly.
  arXiv  Detail & Related papers  (2021-09-28T05:33:21Z)
• Kernel methods through the roof: handling billions of points efficiently [94.31450736250918]
  Kernel methods provide an elegant and principled approach to nonparametric learning, but so far could hardly be used in large scale problems. Recent advances have shown the benefits of a number of algorithmic ideas, for example combining optimization, numerical linear algebra and random projections. Here, we push these efforts further to develop and test a solver that takes full advantage of GPU hardware.
  arXiv  Detail & Related papers  (2020-06-18T08:16:25Z)
• PolyScientist: Automatic Loop Transformations Combined with Microkernels for Optimization of Deep Learning Primitives [55.79741270235602]
  We develop a hybrid solution to the development of deep learning kernels. We use the advanced polyhedral technology to automatically tune the outer loops for performance.
  arXiv  Detail & Related papers  (2020-02-06T08:02:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.