CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging

• URL: http://arxiv.org/abs/2605.08455v1
• Date: Fri, 08 May 2026 20:24:32 GMT
• Title: CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging
• Abstract summary: We introduceBEAVER, a benchmark for evaluating programs produced during real failing workspaces.<n>Each task provides the broken candidate, native build/test commands, raw error evidence, and a single file.<n>We show that protocol-aware evaluation gives a more faithful view of performance-loss tolerance.
• Score: 18.460942231908376
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Debugging CUDA programs has long been challenging because failures often arise from subtle interactions among hardware behavior, compiler decisions, memory hierarchy, and asynchronous execution. More importantly, with the rapid expansion of GPU usage across scientific computing, machine learning, graphics, and systems workloads, CUDA debugging has become more challenging than ever. Current evaluations of LLM-based CUDA programming largely miss this setting: a model can pass correctness tests with repair by degeneration, simplifying the CUDA code into a safer but slower program that abandons the original optimization structure. We introduce CUDABEAVER, a benchmark for CUDA debugging from real failing workspaces produced during LLM-based CUDA generation. Each task provides the broken candidate, native build/test commands, raw error evidence, and a single editable file. CUDABEAVER evaluates whether a fixer truly repairs the failing CUDA code or merely finds a slower test-passing replacement, reporting results by failure category, debugging trajectory, stagnation mode, and performance preservation. We further propose pass@k(M,C,A), a protocol-conditional CUDA debugging metric by making the fixer M, corpus C, and protocol axes Aexplicit. Using this metric across 213 tasks and seven frontier LLMs, we show that protocol-aware evaluation gives a more faithful view of CUDA debugging ability: when performance-loss tolerance is high, fixers appear much stronger, but even a minor stricter performance requirement can sharply reduce measured success, shifting scores by up to 40 percentage points.

Related papers

• CUADebug: Diagnosing and Repairing Computer-Use Agent Failures [65.56066380320107]
  Computer-use agents (CUAs) operate real desktop and web interfaces through screenshots, mouse and keyboard actions, and stateful UI feedback.<n>Unlike text-only agents, CUA failures arise from coupled visual perception, grounding, low-level interaction, task reasoning, and environment dynamics.<n>We introduce CUA Debug, a framework for diagnosing and repairing CUA failures.
  arXiv  Detail & Related papers  (2026-07-31T13:59:45Z)
• Advancing Awkward Arrays for High-Performance CPU and GPU Processing [0.0]
  Awkward Array is a Python library for representing and processing nested, variable-length data.<n>We present developments in the Awkward Array GPU backend, including implementations built on NVIDIA Core Compute Libraries.
  arXiv  Detail & Related papers  (2026-07-30T12:53:25Z)
• GPU Forecasters: Language Models as Selective Surrogates for Kernel Runtime Optimization [73.92934811090163]
  We study how LLMs can serve as selective GPU surrogates for kernel evaluation.<n>We measure whether their forecasts are accurate, calibrated, and practically useful for recovering fast kernels under limited GPU-measurement budgets.<n>Our experiments demonstrate that LLMs can accurately forecast relative kernel performance, that their utility can be improved through reinforcement learning.
  arXiv  Detail & Related papers  (2026-05-29T15:56:08Z)
• CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs [45.52512820646709]
  We introduce Hercules, a benchmark that evaluates generated against end-to-end human-expert SOTA systems.<n>We show that automated programming remains far from fully solved and requires stronger hardware reasoning, better tool use, and training objectives.
  arXiv  Detail & Related papers  (2026-05-08T20:35:27Z)
• CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation [51.72529978689561]
  Agent is a large-scale agentic reinforcement learning system that develops kernel expertise through three components.<n>Agent delivers 100%, 100%, and 92% faster rate over torchcompile on KernelBench.
  arXiv  Detail & Related papers  (2026-02-27T18:58:05Z)
• CUDABench: Benchmarking LLMs for Text-to-CUDA Generation [11.395567633975878]
  Recent studies have demonstrated the potential of Large Language Models (LLMs) in generating GPUs.<n>Given the hardware-specific and performance-critical features of programming, accurately assessing the performance of LLM-generated GPU programs is nontrivial.<n>We introduce GPUBench, a comprehensive benchmark designed to evaluate the text-to-CUDA capabilities of LLMs.
  arXiv  Detail & Related papers  (2026-02-13T09:46:45Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• HPCTransCompile: An AI Compiler Generated Dataset for High-Performance CUDA Transpilation and LLM Preliminary Exploration [13.53425131505526]
  Deep learning has driven exponential increases in model parameters and computational demands.<n> NVIDIA GPUs and their-based software ecosystem provide robust support for parallel computing.<n>The ecosystem has established a dominant position in the field of parallel software.<n> translating code to other platforms poses significant challenges due to differences parallel programming paradigms and hardware.
  arXiv  Detail & Related papers  (2025-06-12T06:48:33Z)
• CUDA-LLM: LLMs Can Write Efficient CUDA Kernels [9.287036563375617]
  Large Language Models (LLMs) have demonstrated strong capabilities in general-purpose code generation.<n>We propose a novel framework called textbfFeature SearchReinforcement (FSR) FSR jointly optimize compilation and functional correctness.
  arXiv  Detail & Related papers  (2025-06-10T10:51:03Z)
• KGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution [59.20933707301566]
  Large Language Models (LLMs) are consistently improving at increasingly realistic software engineering (SE) tasks. In real-world software stacks, significant SE effort is spent developing foundational system software like the Linux kernel. To evaluate if ML models are useful while developing such large-scale systems-level software, we introduce kGym and kBench.
  arXiv  Detail & Related papers  (2024-07-02T21:44:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.