The Correctness Illusion in LLM-Generated GPU Kernels

• URL: http://arxiv.org/abs/2606.20128v1
• Date: Thu, 18 Jun 2026 11:52:46 GMT
• Title: The Correctness Illusion in LLM-Generated GPU Kernels
• Abstract summary: Benchmarks for LLM-generated GPU kernels score correctness through fixed-shape, small-sample allclose-style checks.<n>We construct a controlled corpus of 24 Triton and CPU stand-in kernels.<n>We re-run the same protocol on five GPU classes.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Benchmarks for LLM-generated GPU kernels (KernelBench, TritonBench, GEAK) score correctness through fixed-shape, small-sample allclose-style checks. The number of inputs varies between benchmarks. The shape, dtype, and tolerance are fixed for each kernel. We test that oracle empirically. We construct a controlled corpus of 24 Triton and CPU stand-in kernels (15 correct controls and 9 LLM-style buggy variants seeded with documented transcription errors) and re-evaluate it under op-schema-aware seeded fuzzing with a high-precision (fp64) CPU reference and per-(op, dtype) absolute tolerances. The seeded oracle flags 9 of 9 buggy kernels and passes 15 of 15 correct controls, at zero precision cost on controls. We extend the corpus to 26 ops (adding a flash-attention pair) and re-run the same protocol on five GPU classes (RTX 3060, A10, L40S, A100 SXM4, H100 NVL). The verdicts are identical across all five GPUs: 10 of 10 illusions caught and 16 of 16 controls clean. The corpus result is about LLM-style transcription bugs that the allclose-on-one-shape oracle certifies as correct, not about the bug rate of any specific deployed LLM. Every flagged failure replays byte-for-byte from a stored seed.

Related papers

• AsmEvo: Agentic Assembly-Level Optimization of AMD GPU Kernels with Functional Equivalence Verification [21.700338424307517]
  We present AsmEvo, an agentic assembly-levelaware for AMD GPU kernels.<n>AsmEvo combines code-object recovery, metadata rebuilding, profiling-guided hot-window editing, correctness-gated timing, and patch fallback.<n>On MI308X, AsmEvo improves 29 of 30 selectedBench kernels, reaching 1.35x-mean and 3.88x maximum speedup.
  arXiv  Detail & Related papers  (2026-08-21T03:34:22Z)
• A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family [0.9740025522928777]
  Systems that generate GPU kernels with language models report high correctness rates.<n>Those rates come from a single loose test: run the kernel on a few random inputs at one fixed shape and accept it if the output is close to a reference.<n>We build a contract-grade verifier of twelve adversarial gates, each a property a correct kernel must satisfy, several of them tolerance-free.
  arXiv  Detail & Related papers  (2026-08-13T01:25:56Z)
• Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs [0.0]
  We evaluate seven test-generation strategies across a closeop corpus.<n>We report each strategy on two axes: bug recall and control false-positive (FP) rate.
  arXiv  Detail & Related papers  (2026-06-23T21:43:30Z)
• Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels [0.0]
  tensor- kernel correctness tests go through a fixed-shape all close-style check with hand-picked absolute and relative tolerances.<n>We ask what absolute tolerance would the kernel itself, observed under its correct implementation, justify?<n>The answer is much tighter than the current hand-picked atol.
  arXiv  Detail & Related papers  (2026-06-23T20:50:36Z)
• Auditing Reward Hackability in Code RL Training Environments [0.0]
  We measure the rate at which code RL environments accept incorrect solutions as correct.<n>On a 49-task sample of SWE-bench Verified, 28.5% of tasks have test suites weak enough that a Docker-verified incorrect patch passes them.
  arXiv  Detail & Related papers  (2026-06-14T23:31:42Z)
• GPU Forecasters: Language Models as Selective Surrogates for Kernel Runtime Optimization [73.92934811090163]
  We study how LLMs can serve as selective GPU surrogates for kernel evaluation.<n>We measure whether their forecasts are accurate, calibrated, and practically useful for recovering fast kernels under limited GPU-measurement budgets.<n>Our experiments demonstrate that LLMs can accurately forecast relative kernel performance, that their utility can be improved through reinforcement learning.
  arXiv  Detail & Related papers  (2026-05-29T15:56:08Z)
• WiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systems [0.0]
  We characterize the compilation gap across 17 RepLiQA domains (6,800 questions)<n>We propose WiCER, an iterative algorithm inspired by counterexample-guided abstraction refinement (CEGAR) that closes this gap.<n>An ablation across all 17 topics confirms that targeted diagnosis (+0.95), not generic pinning (+0.16), drives the gains.
  arXiv  Detail & Related papers  (2026-05-08T00:25:16Z)
• Xe-Forge: Multi-Stage LLM-Powered Kernel Optimization for Intel GPU [38.81440160993858]
  Porting deep learning algorithms to new hardware accelerators requires developers to apply the same low-level optimizations to every Triton kernel in their code-base.<n>We present Xe-Forge, a multi-stage LLM-powered pipeline that automates this process for Intel GPU.<n>A curated knowledge base encodes Intel GPU constraints that are absent from LLM training data, keeping the model within architecturally valid bounds.
  arXiv  Detail & Related papers  (2026-04-16T20:21:01Z)
• ARGUS: Agentic GPU Optimization Guided by Data-Flow Invariants [12.49256588033198]
  LLM-based coding agents can generate functionally correct GPU kernels, yet their performance remains far below hand-optimized libraries on critical computations.<n>We present Argus, an agentic framework that addresses this through data-flow invariants.<n>We evaluate Argus on the AMD MI300X GPU across GEMM, flash attention, and MoE kernels.
  arXiv  Detail & Related papers  (2026-04-16T15:49:31Z)
• FlashSampling: Fast and Memory-Efficient Exact Sampling [62.5203057469482]
  FlashSampling is an exact sampling primitive that fuses sampling into the LM-head matmul and never materializes the logits tensor in.<n>Across H100, H200, B200, and B300 GPUs, FlashSampling speeds up kernel-level decode workloads.<n>In end-to-end vLLM experiments, it reduces time per output token by up to $19%$ on the models we test.
  arXiv  Detail & Related papers  (2026-03-16T19:37:08Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• Counting Without Running: Evaluating LLMs' Reasoning About Code Complexity [2.7389338551082605]
  We develop a benchmark to test Large Language Models (LLMs) for anticipating performance bottlenecks.<n>FLOPBench predicts single and double-precision FLOP counts for 577 kernels.<n>Our results positionFLOPBench as a focused testbed for developing LLM tooling.
  arXiv  Detail & Related papers  (2025-12-04T01:03:20Z)
• Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis [57.40527331817245]
  Test oracle generation in non-regression testing is a longstanding challenge in software engineering.<n>We introduce Nexus, a novel multi-agent framework to address this challenge.
  arXiv  Detail & Related papers  (2025-10-30T12:20:25Z)
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference [31.2331188304598]
  Changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses.<n>We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision.<n>Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32.
  arXiv  Detail & Related papers  (2025-06-11T08:23:53Z)
• CLEVER: A Curated Benchmark for Formally Verified Code Generation [53.5486188696892]
  $rm Csmall LEVER$ is a high-quality, curated benchmark of 161 problems for end-to-end verified code generation in Lean.<n>Each problem consists of (1) the task of generating a specification that matches a held-out ground-truth specification, and (2) the task of generating a Lean implementation that provably satisfies this specification.
  arXiv  Detail & Related papers  (2025-05-20T05:15:47Z)
• CrashFixer: A crash resolution agent for the Linux kernel [58.152358195983155]
  This work builds upon kGym, which shares a benchmark for system-level Linux kernel bugs and a platform to run experiments on the Linux kernel.<n>This paper introduces CrashFixer, the first LLM-based software repair agent that is applicable to Linux kernel bugs.
  arXiv  Detail & Related papers  (2025-04-29T04:18:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.