Improving Efficiency of GPU Kernel Optimization Agents using a Domain-Specific Language and Speed-of-Light Guidance

• URL: http://arxiv.org/abs/2603.29010v1
• Date: Mon, 30 Mar 2026 21:16:39 GMT
• Title: Improving Efficiency of GPU Kernel Optimization Agents using a Domain-Specific Language and Speed-of-Light Guidance
• Abstract summary: We make two key observations about the abstraction level that agents operate at.<n>We implement these principles in $$CUTLASS, a DSL with a compiler for CUTLASS-backed GPU kernels.<n>We use SOL guidance to estimate headroom and guide optimization trials, deprioritize problems that are near SOL, and flag kernels that game the benchmark.
• Score: 2.6106833542109693
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optimizing GPU kernels with LLM agents is an iterative process over a large design space. Every candidate must be generated, compiled, validated, and profiled, so fewer trials will save both runtime and cost. We make two key observations. First, the abstraction level that agents operate at is important. If it is too low, the LLM wastes reasoning on low-impact details. If it is too high, it may miss important optimization choices. Second, agents cannot easily tell when they reach the point of diminishing returns, wasting resources as they continue searching. These observations motivate two design principles to improve efficiency: (1) a compact domain-specific language (DSL) that can be learned in context and lets the model reason at a higher level while preserving important optimization levers, and (2) Speed-of-Light (SOL) guidance that uses first-principles performance bounds to steer and budget search. We implement these principles in $μ$CUTLASS, a DSL with a compiler for CUTLASS-backed GPU kernels that covers kernel configuration, epilogue fusion, and multi-stage pipelines. We use SOL guidance to estimate headroom and guide optimization trials, deprioritize problems that are near SOL, and flag kernels that game the benchmark. On 59 KernelBench problems with the same iteration budgets, switching from generating low-level code to DSL code using GPT-5-mini turns a 0.40x geomean regression into a 1.27x speedup over PyTorch. Adding SOL-guided steering raises this to 1.56x. Across model tiers, $μ$CUTLASS + SOL-guidance lets weaker models outperform stronger baseline agents at lower token cost. SOL-guided budgeting saves 19-43% of tokens while retaining at least 95% of geomean speedup, with the best policy reaching a 1.68x efficiency gain. Lastly, SOL analysis helps detect benchmark-gaming cases, where kernels may appear fast while failing to perform the intended computation.

Related papers

• LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization [7.393546440833148]
  Large language models have become increasingly capable agents for low-level code and kernel optimization.<n>We introduce LLM4LLM, a deployment-aware closed-loop optimization framework.<n>It improves end-to-end latency for every evaluated model, achieving 3.91$times$/6.98$times$ geometric-mean speedups on A100/H100.
  arXiv  Detail & Related papers  (2026-08-22T08:12:16Z)
• Compiler-Grounded Hierarchical Diagnosis for LLM-Based Triton Kernel Optimization [10.855607995195099]
  We present a compiler-grounded and hierarchical optimization framework for Triton kernels.<n>The system attains a geometric-mean speedup of 4.35$times$ and a median speedup of 2.73$times$ from the initial optimized to Triton kernel.<n>The complete distribution ranges from near-baseline entries to large wins, motivating transparent reporting of the current system's scope and limitations.
  arXiv  Detail & Related papers  (2026-07-25T07:49:38Z)
• GPU Forecasters: Language Models as Selective Surrogates for Kernel Runtime Optimization [73.92934811090163]
  We study how LLMs can serve as selective GPU surrogates for kernel evaluation.<n>We measure whether their forecasts are accurate, calibrated, and practically useful for recovering fast kernels under limited GPU-measurement budgets.<n>Our experiments demonstrate that LLMs can accurately forecast relative kernel performance, that their utility can be improved through reinforcement learning.
  arXiv  Detail & Related papers  (2026-05-29T15:56:08Z)
• optimize_anything: A Universal API for Optimizing any Text Parameter [98.42497715725356]
  We show a single AI-based optimization system-supporting single-task search, multi-task search with cross-problem transfer, and generalization to unseen inputs.<n>We show for the first time that text optimization with LLM-based search is a general-purpose problem-solving paradigm.
  arXiv  Detail & Related papers  (2026-05-19T10:18:12Z)
• Agent Factories for High Level Synthesis: How Far Can General-Purpose Coding Agents Go in Hardware Optimization? [8.899459735174174]
  We present an empirical study of how far general-purpose coding agents can optimize hardware designs from high-level algorithmic specifications.<n>We introduce an agent factory, a two-stage pipeline that constructs and coordinates multiple autonomous optimization agents.<n>We evaluate the approach on 12 kernels from HLS-Eval and Rodinia-HLS using Claude Code (Opus4.5/4.6) with AMD Vitis HLS.
  arXiv  Detail & Related papers  (2026-03-26T17:57:50Z)
• Resource-Efficient Iterative LLM-Based NAS with Feedback Memory [49.44875022114861]
  Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources.<n>We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures.
  arXiv  Detail & Related papers  (2026-03-12T16:00:22Z)
• KernelSkill: A Multi-Agent Framework for GPU Kernel Optimization [37.008713142905336]
  We present KernelSkill, a multi-agent framework with a dual-level memory architecture.<n>On Kernel Levels 1-3, KernelSkill achieves a 100% success rate and average speedups of 5.44xBench, 2.82x, and 1.92x over Torch Eager.
  arXiv  Detail & Related papers  (2026-03-10T13:43:38Z)
• $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• A Two-Stage GPU Kernel Tuner Combining Semantic Refactoring and Search-Based Optimization [9.49293344824955]
  This paper introduces a template-based rewriting layer on top of an agent-driven iterative loop.<n>The proposed method can be extended to deliver automated performance optimization for real production workloads.
  arXiv  Detail & Related papers  (2026-01-19T03:40:12Z)
• AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimization [12.269456144158783]
  We present AccelOpt, a self-improving large language model (LLM) agentic system that autonomously optimize kernels for emerging AI acclerators.<n>We build NKIBench, a new benchmark suite of AWS Trainium accelerator kernels with varying complexity extracted from real-world workloads.<n>Our evaluation confirms that AccelOpt's capability improves over time, boosting the average percentage of peak throughput from $49%$ to $61%$ on Trainium 1 and from $45%$ to $59%$ on NKIBench kernels.
  arXiv  Detail & Related papers  (2025-11-19T22:49:37Z)
• Controlling Thinking Speed in Reasoning Models [57.14541748751654]
  Human cognition operates in two modes: fast, intuitive System 1 thinking and slow, deliberate System 2 thinking.<n>In this work, we enable LRMs to approximate human intelligence through dynamic thinking speed adjustment.<n>Our approach addresses two key questions: (1) how to control thinking speed in LRMs, and (2) when to adjust it for optimal performance.
  arXiv  Detail & Related papers  (2025-07-04T16:41:06Z)
• Second-order Optimization of Gaussian Splats with Importance Sampling [51.95046424364725]
  3D Gaussian Splatting (3DGS) is widely used for novel view rendering due to its high quality and fast inference time.<n>We propose a novel second-order optimization strategy based on Levenberg-Marquardt (LM) and Conjugate Gradient (CG)<n>Our method achieves a $3times$ speedup over standard LM and outperforms Adam by $6times$ when the Gaussian count is low.
  arXiv  Detail & Related papers  (2025-04-17T12:52:08Z)
• Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement [47.89758553708932]
  We introduce textbfThinkCoder, a framework that combines thorough exploration with optimal refinement.<n>The exploration phase diversifies the solution space by searching for potential solutions, followed by a refinement phase that enhances precision.<n>To further minimize test-time computation overhead, we introduce preference-driven optimization with Reinforced Self-Training (ReST)
  arXiv  Detail & Related papers  (2024-12-30T07:02:15Z)
• Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient [57.9629676017527]
  We propose an optimization-based structural pruning that learns the pruning masks in a probabilistic space directly by optimizing the loss of the pruned model.<n>We achieve this by learning an underlying Bernoulli distribution to sample binary pruning masks.<n>Experiments conducted on LLaMA, LLaMA-2, LLaMA-3, Vicuna, and Mistral models demonstrate the promising performance of our method in efficiency and effectiveness.
  arXiv  Detail & Related papers  (2024-06-15T09:31:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.