Correct but Slow: An Empirical Study of the GPU Kernel Evaluation Gap in Modern Domain-Specific Languages

• URL: http://arxiv.org/abs/2607.04454v3
• Date: Mon, 13 Jul 2026 19:43:36 GMT
• Title: Correct but Slow: An Empirical Study of the GPU Kernel Evaluation Gap in Modern Domain-Specific Languages
• Abstract summary: A functionally valid kernel may still fall far below the throughput of the optimized library operator it is intended to replace.<n>We study this correctness-performance gap using 22 Triton and TileLang kernels from five operator categories on NVIDIA A100 and GH200 GPUs.
• Score: 9.908756853186018
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Modern GPU domain-specific languages (DSLs), such as Triton and TileLang, are increasingly used to implement specialized deep-learning kernels and as target languages for automated kernel-generation systems. Existing DSL-kernel evaluations establish correctness through reference-based numerical validation -- necessary, but silent on replacement quality: a functionally valid kernel may still fall far below the throughput of the optimized library operator it is intended to replace. We study this correctness-performance gap using 22 Triton and TileLang kernels from five operator categories on NVIDIA A100 and GH200 GPUs, asking whether correctness-based evaluation identifies kernels unsuitable as library replacements, why such failures occur, and how they can be detected without exhaustive benchmark coverage. The study yields three results. \emph{First}, correctness-based evaluation can admit severe slowdowns: an idiomatic TileLang LayerNorm kernel passes KernelBench's correctness check while running more than 300$\times$ slower than the PyTorch baseline. \emph{Second}, the causes differ by kernel family. TileLang normalization and reduction slowdowns are mainly repairable authoring defects, such as sequential reductions and unnecessary dtype conversions, whereas convolution and large general matrix multiplication (GEMM) retain residual gaps after optimization due to code-generation and autotuning-coverage limits; vendor-library algorithm selection contributes only marginally. \emph{Third}, two lightweight checks -- library-relative efficiency and roofline utilization -- are complementary screening criteria: together they flag every functionally valid but inefficient kernel in our suite and separate repairable authoring defects from structural residuals.

Related papers

• KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch? [13.026983194416168]
  Large language models (LLMs) can generate custom kernels that appear to outperform PyTorch on benchmarks such as KernelBench.<n>We demonstrate that frontier models frequently engage in reward hacking to artificially inflate reported performance.
  arXiv  Detail & Related papers  (2026-06-26T03:22:50Z)
• GPU Forecasters: Language Models as Selective Surrogates for Kernel Runtime Optimization [73.92934811090163]
  We study how LLMs can serve as selective GPU surrogates for kernel evaluation.<n>We measure whether their forecasts are accurate, calibrated, and practically useful for recovering fast kernels under limited GPU-measurement budgets.<n>Our experiments demonstrate that LLMs can accurately forecast relative kernel performance, that their utility can be improved through reinforcement learning.
  arXiv  Detail & Related papers  (2026-05-29T15:56:08Z)
• Kernel Foundry: A Diagnosis-driven Evolutionary Kernel Optimizer with Multi-Experts [29.878782850560132]
  Kernel Foundry is a diagnosis-driven evolutionary framework for automatic GPU kernel optimization.<n> Experiments on KernelBench show that our method consistently improves both correctness and performance over strong baselines.
  arXiv  Detail & Related papers  (2026-05-08T03:41:54Z)
• AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation [59.964989458924585]
  AdaExplore is an agent framework that enables self-improvement via accumulated execution feedback for performance-critical kernel code generation.<n>In the adaptation stage, the agent synthesizes tasks and converts recurring failures into a reusable memory of validity rules.<n>In the search stage, the agent organizes candidate kernels as a tree and alternates between small local refinements and larger structural regeneration.
  arXiv  Detail & Related papers  (2026-04-17T18:25:03Z)
• Xe-Forge: Multi-Stage LLM-Powered Kernel Optimization for Intel GPU [38.81440160993858]
  Porting deep learning algorithms to new hardware accelerators requires developers to apply the same low-level optimizations to every Triton kernel in their code-base.<n>We present Xe-Forge, a multi-stage LLM-powered pipeline that automates this process for Intel GPU.<n>A curated knowledge base encodes Intel GPU constraints that are absent from LLM training data, keeping the model within architecturally valid bounds.
  arXiv  Detail & Related papers  (2026-04-16T20:21:01Z)
• ARGUS: Agentic GPU Optimization Guided by Data-Flow Invariants [12.49256588033198]
  LLM-based coding agents can generate functionally correct GPU kernels, yet their performance remains far below hand-optimized libraries on critical computations.<n>We present Argus, an agentic framework that addresses this through data-flow invariants.<n>We evaluate Argus on the AMD MI300X GPU across GEMM, flash attention, and MoE kernels.
  arXiv  Detail & Related papers  (2026-04-16T15:49:31Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• CrashFixer: A crash resolution agent for the Linux kernel [58.152358195983155]
  This work builds upon kGym, which shares a benchmark for system-level Linux kernel bugs and a platform to run experiments on the Linux kernel.<n>This paper introduces CrashFixer, the first LLM-based software repair agent that is applicable to Linux kernel bugs.
  arXiv  Detail & Related papers  (2025-04-29T04:18:51Z)
• KernelBench: Can LLMs Write Efficient GPU Kernels? [36.4117525096377]
  KernelBench is an open-source framework for evaluating language models' ability to write fast and correct kernels.<n>We introduce a new evaluation metric fast_p, which measures the percentage of generated kernels that are functionally correct.<n>Our experiments show that frontier reasoning models perform the best out of the box but still fall short overall.
  arXiv  Detail & Related papers  (2025-02-14T19:30:53Z)
• KGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution [59.20933707301566]
  Large Language Models (LLMs) are consistently improving at increasingly realistic software engineering (SE) tasks. In real-world software stacks, significant SE effort is spent developing foundational system software like the Linux kernel. To evaluate if ML models are useful while developing such large-scale systems-level software, we introduce kGym and kBench.
  arXiv  Detail & Related papers  (2024-07-02T21:44:22Z)
• Lifelong Bandit Optimization: No Prior and No Regret [70.94238868711952]
  We develop LIBO, an algorithm which adapts to the environment by learning from past experience. We assume a kernelized structure where the kernel is unknown but shared across all tasks. Our algorithm can be paired with any kernelized or linear bandit algorithm and guarantees optimal performance.
  arXiv  Detail & Related papers  (2022-10-27T14:48:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.