KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware

• URL: http://arxiv.org/abs/2603.08721v1
• Date: Tue, 10 Feb 2026 14:52:02 GMT
• Title: KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware
• Abstract summary: New AI accelerators often require developers to manually craft low-level kernels.<n>This prevents emerging hardware platforms from reaching the market efficiently.<n>We present KernelCraft: the first benchmark to evaluate an agent's ability to generate and optimize low-level kernels for customized accelerators.
• Score: 25.808580418841718
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: New AI accelerators with novel instruction set architectures (ISAs) often require developers to manually craft low-level kernels -- a time-consuming, laborious, and error-prone process that cannot scale across diverse hardware targets. This prevents emerging hardware platforms from reaching the market efficiently. While prior LLM-based code generation has shown promise in mature GPU ecosystems, it remains unclear whether agentic LLM systems can quickly produce valid and efficient kernels for emerging hardware with new ISAs. We present KernelCraft: the first benchmark to evaluate an LLM agent's ability to generate and optimize low-level kernels for customized accelerators via a function-calling, feedback-driven workflow. Within KernelCraft, the agent refines kernels under ISA and hardware constraints using automated feedback derived from compilation checks, simulation, and correctness validation against ground truth. In our experiments, we assess agent performance across three emerging accelerator platforms on more than 20 ML tasks, each with 5 diverse task configurations, with special evaluation of task configuration complexity. Across four leading reasoning models, top agents produce functionally valid kernels for previously unseen ISAs within a few refinement steps, with optimized kernels that match or outperform template-based compiler baselines. With that, we demonstrate the potential for reducing the cost of kernel development for accelerator designers and kernel developers.

Related papers

• FastKernels: Benchmarking GPU Kernel Generation in Production [15.284289018908781]
  LLM-based agents for GPU kernel generation are advancing rapidly.<n>Yet their progress is constrained by the benchmarks they optimize against.<n>Fast Kernels is a production-grade inference framework that runs at parity with hardened systems.
  arXiv  Detail & Related papers  (2026-05-22T04:19:04Z)
• AutoVecCoder: Teaching LLMs to Generate Explicitly Vectorized Code [69.84692701646769]
  AutoVecCoder is a novel framework designed to empower Large Language Models with the capability of automated explicit vectorization.<n>VecPrompt, an automated data synthesis pipeline, injects domain-specific intrinsic knowledge.<n>VecRL, a reinforcement learning framework, aligns code generation with execution efficiency.<n>AutoVecCoder-8B trained by this framework achieves state-of-the-art performance on the SSE and AVX subsets of SimdBench.
  arXiv  Detail & Related papers  (2026-05-18T07:33:15Z)
• Kernel-Smith: A Unified Recipe for Evolutionary Kernel Optimization [48.656549870801285]
  Kernel-Smith is a framework for high-performance GPU kernel and operator generation.<n>On the agent side, Kernel-Smith maintains a population of executable candidates and iteratively improves them.<n>On the training side, we convert long-horizon evolution trajectories into step-centric supervision and reinforcement learning signals.
  arXiv  Detail & Related papers  (2026-03-30T12:12:49Z)
• Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesis [68.7701048879757]
  EvoKernel is a self-evolving agentic framework that automates the lifecycle of kernel synthesis.<n>It learns stage-specific Q-values that prioritize experiences based on their contribution to the current objective.<n>It improves models' correctness from 11.0% to 83.0% and a median speedup of 3.60x over initial drafts.
  arXiv  Detail & Related papers  (2026-03-11T14:57:06Z)
• AscendCraft: Automatic Ascend NPU Kernel Generation via DSL-Guided Transcompilation [8.878393510726008]
  We present AscendCraft, a DSL-guided approach for automatic AscendC kernel generation.<n>AscendingCraft achieves 98.1% compilation success and 90.4% functional correctness.<n>We also demonstrate that DSL-guided transcompilation can enable LLMs to generate both correct and competitive NPU kernels.
  arXiv  Detail & Related papers  (2026-01-30T09:34:59Z)
• Towards Automated Kernel Generation in the Era of LLMs [17.69471168609145]
  Kernel engineering is a time-consuming and non-scalable process.<n>Recent advances in large language models (LLMs) and agentic systems have opened new possibilities for automating kernel generation and optimization.<n>The field remains fragmented, lacking a systematic perspective for LLM-driven kernel generation.
  arXiv  Detail & Related papers  (2026-01-22T07:53:52Z)
• AKG kernel Agent: A Multi-Agent Framework for Cross-Platform Kernel Synthesis [13.239454996851771]
  Modern AI models demand high-performance computation kernels.<n>Akg kernel agent (AI-driven Kernel Generator) is designed to support multiple domain-specific languages.<n>System's modular design allows rapid integration of backend DSLs and hardware targets.<n>System achieves an average speedup of 1.46$times over PyTorch Eager baselines.
  arXiv  Detail & Related papers  (2025-12-29T12:42:05Z)
• STARK: Strategic Team of Agents for Refining Kernels [23.717055490630596]
  We introduce an agentic framework for GPU kernel optimization that explores the design space through multi-agent collaboration.<n>This framework mimics the workflow of expert engineers, enabling LLMs to reason about hardware trade-offs, incorporate profiling feedback, and refine kernels iteratively.<n>We evaluate our approach on KernelBench, a benchmark for LLM-based kernel optimization, and demonstrate substantial improvements over baseline agents.
  arXiv  Detail & Related papers  (2025-10-19T20:41:46Z)
• MigGPT: Harnessing Large Language Models for Automated Migration of Out-of-Tree Linux Kernel Patches Across Versions [53.811953357289866]
  Large language models (LLMs) have shown remarkable progress across various domains.<n>LLMs struggle with incomplete code context understanding and inaccurate migration point identification.<n>MigGPT is a framework that employs a novel code fingerprint structure to retain code snippet information.
  arXiv  Detail & Related papers  (2025-04-13T08:08:37Z)
• Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures [67.47328776279204]
  This work introduces a framework to develop efficient, portable Deep Learning and High Performance Computing kernels. We decompose the kernel development in two steps: 1) Expressing the computational core using Processing Primitives (TPPs) and 2) Expressing the logical loops around TPPs in a high-level, declarative fashion. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
  arXiv  Detail & Related papers  (2023-04-25T05:04:44Z)
• PolyScientist: Automatic Loop Transformations Combined with Microkernels for Optimization of Deep Learning Primitives [55.79741270235602]
  We develop a hybrid solution to the development of deep learning kernels. We use the advanced polyhedral technology to automatically tune the outer loops for performance.
  arXiv  Detail & Related papers  (2020-02-06T08:02:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.