Learning When to Optimize: Verified Optimization Skills from Expert GPU-Kernel Lineages

• URL: http://arxiv.org/abs/2605.28213v1
• Date: Wed, 27 May 2026 09:32:16 GMT
• Title: Learning When to Optimize: Verified Optimization Skills from Expert GPU-Kernel Lineages
• Abstract summary: LLM-based agents are increasingly used to generate GPU kernels.<n>They often know what optimizations to try without knowing when those optimizations are sound.<n>We introduce KLineage, which learns this missing "when" knowledge from expert kernels.
• Score: 4.0702755941542925
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-based agents are increasingly used to generate GPU kernels, but they often know what optimizations to try without knowing when those optimizations are sound. We introduce KLineage, which learns this missing "when" knowledge from expert kernels: instead of relying on forward rollouts, KLineage walks expert implementations backward through validation-gated simplifications and reverses each accepted step into a reusable optimization skill. Each skill records not only the optimization intent, but also where it applies in code, what conditions made it valid, what effect it had, and what failures its assumptions avoid. A downstream LLM materializes these skills on new code surfaces under the same compile/correctness/profile gate. On five expert workloads across two NVIDIA architectures, these lineage-derived skills serve as an effective optimization curriculum, exceeding recent memory-based LLM-kernel baselines in both final kernel quality and optimization efficiency under the same fixed budget. We additionally use a separate 22-instance held-out check as a sanity test against source-case memorization.

Related papers

• HIERA: Workload-Aware Planning Across Implementation Spaces for GPU Kernel Optimization [17.600449573236194]
  We propose textscHIERA, a hierarchical search-space planning framework for kernel optimization.<n>textscHIERA constructs contract-augmented task specifications, selects an appropriate implementation space across PyTorch operators, libraries, and custom kernels.<n>Experiments show textscHIERA delivers stronger overall implementation refinement, validity, and optimization performance than existing training-free methods.
  arXiv  Detail & Related papers  (2026-08-21T14:29:20Z)
• Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization [20.018878620048344]
  In pure black-box optimization, LLMs act as greedy greedys.<n>In zero-shot kernel generation, providing explicit input-size information has no measurable effect.<n>When tasked to perform optimization for uncommon kernel sizes, performance sharply degrades regardless of the language used.
  arXiv  Detail & Related papers  (2026-05-19T12:48:48Z)
• Learn2Splat: Extending the Horizon of Learned 3DGS Optimization [53.913063313338874]
  We introduce a learned for 3DGS that avoids degradation over extended horizons without auxiliary mechanisms.<n>Results improved novel view quality while remaining stable over long horizons, with zero-shot generalization to unseen reconstruction settings.
  arXiv  Detail & Related papers  (2026-05-15T09:21:29Z)
• PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization [43.72919370258115]
  We introduce PerfCodeBench, an executable benchmark for evaluating large language models (LLMs) on high-performance code optimization.<n>The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenecks.<n>Our evaluation shows a clear gap between model-generated code and expert-optimized implementations.
  arXiv  Detail & Related papers  (2026-05-13T08:10:26Z)
• KernelSkill: A Multi-Agent Framework for GPU Kernel Optimization [37.008713142905336]
  We present KernelSkill, a multi-agent framework with a dual-level memory architecture.<n>On Kernel Levels 1-3, KernelSkill achieves a 100% success rate and average speedups of 5.44xBench, 2.82x, and 1.92x over Torch Eager.
  arXiv  Detail & Related papers  (2026-03-10T13:43:38Z)
• Making LLMs Optimize Multi-Scenario CUDA Kernels Like Experts [25.264078143000045]
  Current automated kernel optimization methods focus on machine learning applications, while overlooking broader domains like sparse matrix operations in scientific computing.<n>In this paper, we address the absence of systematic evaluation for multi-scenario settings introducing MSBench.<n>Building on this benchmark, we introduce MSMaster, a hardware-aware system for kernel optimization that leverages GPU profiling information and automatically constructs the full compilation and execution toolchain.
  arXiv  Detail & Related papers  (2026-03-07T12:22:43Z)
• LLM as a Complementary Optimizer to Gradient Descent: A Case Study in Prompt Tuning [69.95292905263393]
  We show that gradient-based and high-level LLMs can effectively collaborate a combined optimization framework.<n>In this paper, we show that these complementary to each other and can effectively collaborate a combined optimization framework.
  arXiv  Detail & Related papers  (2024-05-30T06:24:14Z)
• Learning to Generalize Provably in Learning to Optimize [185.71326306329678]
  Learning to optimize (L2O) has gained increasing popularity, which automates the design of optimizees by data-driven approaches. Current L2O methods often suffer from poor generalization performance in at least two folds. We propose to incorporate these two metrics as flatness-aware regularizers into the L2O framework.
  arXiv  Detail & Related papers  (2023-02-22T01:17:31Z)
• VeLO: Training Versatile Learned Optimizers by Scaling Up [67.90237498659397]
  We leverage the same scaling approach behind the success of deep learning to learn versatiles. We train an ingest for deep learning which is itself a small neural network that ingests and outputs parameter updates. We open source our learned, meta-training code, the associated train test data, and an extensive benchmark suite with baselines at velo-code.io.
  arXiv  Detail & Related papers  (2022-11-17T18:39:07Z)
• Learning to Optimize: A Primer and A Benchmark [94.29436694770953]
  Learning to optimize (L2O) is an emerging approach that leverages machine learning to develop optimization methods. This article is poised to be the first comprehensive survey and benchmark of L2O for continuous optimization.
  arXiv  Detail & Related papers  (2021-03-23T20:46:20Z)
• Reverse engineering learned optimizers reveals known and novel mechanisms [50.50540910474342]
  Learneds are algorithms that can themselves be trained to solve optimization problems. Our results help elucidate the previously murky understanding of how learneds work, and establish tools for interpreting future learneds.
  arXiv  Detail & Related papers  (2020-11-04T07:12:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.