AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimization

• URL: http://arxiv.org/abs/2511.15915v1
• Date: Wed, 19 Nov 2025 22:49:37 GMT
• Title: AccelOpt: A Self-Improving LLM Agentic System for AI Accelerator Kernel Optimization
• Abstract summary: We present AccelOpt, a self-improving large language model (LLM) agentic system that autonomously optimize kernels for emerging AI acclerators.<n>We build NKIBench, a new benchmark suite of AWS Trainium accelerator kernels with varying complexity extracted from real-world workloads.<n>Our evaluation confirms that AccelOpt's capability improves over time, boosting the average percentage of peak throughput from $49%$ to $61%$ on Trainium 1 and from $45%$ to $59%$ on NKIBench kernels.
• Score: 12.269456144158783
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present AccelOpt, a self-improving large language model (LLM) agentic system that autonomously optimizes kernels for emerging AI acclerators, eliminating the need for expert-provided hardware-specific optimization knowledge. AccelOpt explores the kernel optimization space through iterative generation, informed by an optimization memory that curates experiences and insights from previously encountered slow-fast kernel pairs. We build NKIBench, a new benchmark suite of AWS Trainium accelerator kernels with varying complexity extracted from real-world LLM workloads to evaluate the effectiveness of AccelOpt. Our evaluation confirms that AccelOpt's capability improves over time, boosting the average percentage of peak throughput from $49\%$ to $61\%$ on Trainium 1 and from $45\%$ to $59\%$ on Trainium 2 for NKIBench kernels. Moreover, AccelOpt is highly cost-effective: using open-source models, it matches the kernel improvements of Claude Sonnet 4 while being $26\times$ cheaper.

Related papers

• LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization [7.393546440833148]
  Large language models have become increasingly capable agents for low-level code and kernel optimization.<n>We introduce LLM4LLM, a deployment-aware closed-loop optimization framework.<n>It improves end-to-end latency for every evaluated model, achieving 3.91$times$/6.98$times$ geometric-mean speedups on A100/H100.
  arXiv  Detail & Related papers  (2026-08-22T08:12:16Z)
• Compiler-Grounded Hierarchical Diagnosis for LLM-Based Triton Kernel Optimization [10.855607995195099]
  We present a compiler-grounded and hierarchical optimization framework for Triton kernels.<n>The system attains a geometric-mean speedup of 4.35$times$ and a median speedup of 2.73$times$ from the initial optimized to Triton kernel.<n>The complete distribution ranges from near-baseline entries to large wins, motivating transparent reporting of the current system's scope and limitations.
  arXiv  Detail & Related papers  (2026-07-25T07:49:38Z)
• AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation [59.964989458924585]
  AdaExplore is an agent framework that enables self-improvement via accumulated execution feedback for performance-critical kernel code generation.<n>In the adaptation stage, the agent synthesizes tasks and converts recurring failures into a reusable memory of validity rules.<n>In the search stage, the agent organizes candidate kernels as a tree and alternates between small local refinements and larger structural regeneration.
  arXiv  Detail & Related papers  (2026-04-17T18:25:03Z)
• Improving Efficiency of GPU Kernel Optimization Agents using a Domain-Specific Language and Speed-of-Light Guidance [2.6106833542109693]
  We make two key observations about the abstraction level that agents operate at.<n>We implement these principles in $$CUTLASS, a DSL with a compiler for CUTLASS-backed GPU kernels.<n>We use SOL guidance to estimate headroom and guide optimization trials, deprioritize problems that are near SOL, and flag kernels that game the benchmark.
  arXiv  Detail & Related papers  (2026-03-30T21:16:39Z)
• Kernel-Smith: A Unified Recipe for Evolutionary Kernel Optimization [48.656549870801285]
  Kernel-Smith is a framework for high-performance GPU kernel and operator generation.<n>On the agent side, Kernel-Smith maintains a population of executable candidates and iteratively improves them.<n>On the training side, we convert long-horizon evolution trajectories into step-centric supervision and reinforcement learning signals.
  arXiv  Detail & Related papers  (2026-03-30T12:12:49Z)
• AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization [13.951709632040908]
  We present Ascendr, an episodic agent that bootstraps this missing expertise by turning execution into experience.<n>Ascendr achieves a 1.19x geometric-mean speedup over the open-source baseline, with 49.61% of operators their references, outperforming strong agent and search baselines.
  arXiv  Detail & Related papers  (2026-03-24T08:54:53Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• KernelSkill: A Multi-Agent Framework for GPU Kernel Optimization [37.008713142905336]
  We present KernelSkill, a multi-agent framework with a dual-level memory architecture.<n>On Kernel Levels 1-3, KernelSkill achieves a 100% success rate and average speedups of 5.44xBench, 2.82x, and 1.92x over Torch Eager.
  arXiv  Detail & Related papers  (2026-03-10T13:43:38Z)
• CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation [51.72529978689561]
  Agent is a large-scale agentic reinforcement learning system that develops kernel expertise through three components.<n>Agent delivers 100%, 100%, and 92% faster rate over torchcompile on KernelBench.
  arXiv  Detail & Related papers  (2026-02-27T18:58:05Z)
• Dr. Kernel: Reinforcement Learning Done Right for Triton Kernel Generations [32.98036846113632]
  We study reinforcement learning (RL) for kernel generation.<n>We propose Turn-level Reinforce-Leave-One-Out (TRLOO) to provide unbiased advantage estimation.<n>We introduce Profiling-based Rewards (PR) and Profiling-based Rejection Sampling (PRS) to overcome the issue.
  arXiv  Detail & Related papers  (2026-02-05T17:01:09Z)
• Evolutionary Mapping of Neural Networks to Spatial Accelerators [64.13809409887254]
  We introduce the first evolutionary, hardware-in-the-loop mapping framework for neuromorphic accelerators.<n>We evaluate our approach on Intel Loihi 2, a representative spatial accelerator featuring 152 cores in a 2D mesh.<n>Our method achieves up to 35% reduction in total latency compared to default cores on two sparse multi-layer perceptron networks.
  arXiv  Detail & Related papers  (2026-02-04T16:28:08Z)
• AscendCraft: Automatic Ascend NPU Kernel Generation via DSL-Guided Transcompilation [8.878393510726008]
  We present AscendCraft, a DSL-guided approach for automatic AscendC kernel generation.<n>AscendingCraft achieves 98.1% compilation success and 90.4% functional correctness.<n>We also demonstrate that DSL-guided transcompilation can enable LLMs to generate both correct and competitive NPU kernels.
  arXiv  Detail & Related papers  (2026-01-30T09:34:59Z)
• SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations [54.303301888915406]
  Mixture of Experts (MoE) models have emerged as the de facto architecture for scaling up language models without significantly increasing the computational cost.<n>We propose a memory-efficient algorithm to compute the forward and backward passes of MoEs with minimal activation caching.<n>We also propose a novel "token rounding" method that minimizes the wasted compute due to padding in Grouped GEMM kernels.
  arXiv  Detail & Related papers  (2025-12-16T04:39:10Z)
• CudaForge: An Agent Framework with Hardware Feedback for CUDA Kernel Optimization [36.794824560677064]
  CudaForge is a training-free multi-agent workflow for kernel generation and optimization.<n>By leveraging base models like OpenAI-o3, CudaForge achieves 97.6% correctness generated kernels and an average 1.68$times$ speedup.
  arXiv  Detail & Related papers  (2025-10-23T22:52:00Z)
• STARK: Strategic Team of Agents for Refining Kernels [23.717055490630596]
  We introduce an agentic framework for GPU kernel optimization that explores the design space through multi-agent collaboration.<n>This framework mimics the workflow of expert engineers, enabling LLMs to reason about hardware trade-offs, incorporate profiling feedback, and refine kernels iteratively.<n>We evaluate our approach on KernelBench, a benchmark for LLM-based kernel optimization, and demonstrate substantial improvements over baseline agents.
  arXiv  Detail & Related papers  (2025-10-19T20:41:46Z)
• EvoEngineer: Mastering Automated CUDA Kernel Code Evolution with Large Language Models [27.430839306140157]
  Large Language Models (LLMs) for automating kernel optimization promise promise.<n>General-purpose LLM code evolution methods cannot meet strict correctness requirements of kernel optimization.<n>EvoEngineer provides guidance for designing and adapting optimization strategies to achieve a balance between performance and correctness.<n>Our method achieves a maximum speedup of textbf36.75$times among all operations over PyTorch kernels and delivers the highest speedup on textbf28 (textbf56.0%) of 50 operations that achieve over textbf2times$ acceleration.
  arXiv  Detail & Related papers  (2025-10-04T10:00:25Z)
• Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval [67.21678698740267]
  We introduce Spotlight Attention, a novel method that employs non-linear hashing functions to optimize the embedding distribution of queries and keys.<n>We also develop a lightweight, stable training framework using a Bradley-Terry ranking-based loss.
  arXiv  Detail & Related papers  (2025-08-27T10:11:27Z)
• Liger Kernel: Efficient Triton Kernels for LLM Training [6.373771349397682]
  Training Large Language Models (LLMs) efficiently at scale presents a formidable challenge, driven by their ever-increasing computational demands.<n>We introduce Liger- Kernel, an open-sourced set of Triton kernels developed specifically for LLM training.<n>With kernel optimization techniques like kernel operation fusing and input chunking, our kernels achieve on average a 20% increase in training throughput and a 60% reduction in GPU memory usage.
  arXiv  Detail & Related papers  (2024-10-14T18:17:01Z)
• Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference [23.633481089469836]
  Auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance.<n>We propose a novel parallel prompt decoding that requires only $0.0002$% trainable parameters, enabling efficient training on a single A100-40GB GPU in just 16 hours.<n>Our approach demonstrates up to 2.49$times$ speedup and maintains a minimal memory overhead of just $0.0004$%.
  arXiv  Detail & Related papers  (2024-05-28T22:19:30Z)
• Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures [67.47328776279204]
  This work introduces a framework to develop efficient, portable Deep Learning and High Performance Computing kernels. We decompose the kernel development in two steps: 1) Expressing the computational core using Processing Primitives (TPPs) and 2) Expressing the logical loops around TPPs in a high-level, declarative fashion. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
  arXiv  Detail & Related papers  (2023-04-25T05:04:44Z)
• PolyScientist: Automatic Loop Transformations Combined with Microkernels for Optimization of Deep Learning Primitives [55.79741270235602]
  We develop a hybrid solution to the development of deep learning kernels. We use the advanced polyhedral technology to automatically tune the outer loops for performance.
  arXiv  Detail & Related papers  (2020-02-06T08:02:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.