LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization

• URL: http://arxiv.org/abs/2608.21836v1
• Date: Sat, 22 Aug 2026 08:12:16 GMT
• Title: LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization
• Abstract summary: Large language models have become increasingly capable agents for low-level code and kernel optimization.<n>We introduce LLM4LLM, a deployment-aware closed-loop optimization framework.<n>It improves end-to-end latency for every evaluated model, achieving 3.91$times$/6.98$times$ geometric-mean speedups on A100/H100.
• Score: 7.393546440833148
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models have become increasingly capable agents for low-level code and kernel optimization, but isolated kernel benchmarks provide only a proxy for the deployment behavior that matters in language-model inference. We identify a benchmark-to-deployment gap: candidate kernels that appear correct and fast in standalone harnesses can exhibit different performance, safety, or phase behavior after integration into a real inference workload. We introduce LLM4LLM, a deployment-aware closed-loop optimization framework that starts from a target inference script, extracts phase-aware optimization tasks, searches with an experience-guided episodic agent, and accepts patches through in-model validation. Across ten language-model inference workloads on A100 and H100 GPUs, LLM4LLM improves end-to-end latency for every evaluated model, achieving 3.91$\times$/6.98$\times$ geometric-mean speedups on A100/H100; as supporting kernel-level evidence, it also attains up to 2.745$\times$ GeoMean speedup on KernelBench Level 2.

Related papers

• Beyond Scaling: Self-Evolving LLM Agents for Hardware Kernel Optimization via an Experience-Driven Workflow and Experience Graph Memory [19.236924182307085]
  Existing kernel-optimization agents seldom preserve a decision, its observed execution feedback, and the later decisions that use that evidence.<n>We present KOPE, an experience-driven framework for hardware kernel optimization.
  arXiv  Detail & Related papers  (2026-08-26T09:27:39Z)
• PACE: A Proxy for Agentic Capability Evaluation [60.3743414796937]
  We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations.<n>Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost.
  arXiv  Detail & Related papers  (2026-07-02T10:59:03Z)
• KForge: LLM-Driven Cross-Platform Kernel Generation for AI Accelerators [5.703659894138959]
  Production inference increasingly targets a heterogeneous mix of accelerators.<n>This creates a systems challenge: each pipeline now requires high-performance kernels across a growing set of hardware backends and programming models.<n>We present KForge, a cross-platform framework built around an iterative refinement loop driven by two collaborating LLM-based agents.
  arXiv  Detail & Related papers  (2026-06-01T23:48:55Z)
• FastKernels: Benchmarking GPU Kernel Generation in Production [15.284289018908781]
  LLM-based agents for GPU kernel generation are advancing rapidly.<n>Yet their progress is constrained by the benchmarks they optimize against.<n>Fast Kernels is a production-grade inference framework that runs at parity with hardened systems.
  arXiv  Detail & Related papers  (2026-05-22T04:19:04Z)
• Optimas: An Intelligent Analytics-Informed Generative AI Framework for Performance Optimization [0.6117371161379209]
  Optimas is a modular, fully automated, end-to-end generative AI framework built on a multi-agent workflow.<n>It maps performance diagnostics to established, literature-backed code transformations.<n>It generates 100% correct code and improves performance in over 98.82% of experiments.
  arXiv  Detail & Related papers  (2026-04-26T21:34:51Z)
• AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation [59.964989458924585]
  AdaExplore is an agent framework that enables self-improvement via accumulated execution feedback for performance-critical kernel code generation.<n>In the adaptation stage, the agent synthesizes tasks and converts recurring failures into a reusable memory of validity rules.<n>In the search stage, the agent organizes candidate kernels as a tree and alternates between small local refinements and larger structural regeneration.
  arXiv  Detail & Related papers  (2026-04-17T18:25:03Z)
• Improving Efficiency of GPU Kernel Optimization Agents using a Domain-Specific Language and Speed-of-Light Guidance [2.6106833542109693]
  We make two key observations about the abstraction level that agents operate at.<n>We implement these principles in $$CUTLASS, a DSL with a compiler for CUTLASS-backed GPU kernels.<n>We use SOL guidance to estimate headroom and guide optimization trials, deprioritize problems that are near SOL, and flag kernels that game the benchmark.
  arXiv  Detail & Related papers  (2026-03-30T21:16:39Z)
• Agent Factories for High Level Synthesis: How Far Can General-Purpose Coding Agents Go in Hardware Optimization? [8.899459735174174]
  We present an empirical study of how far general-purpose coding agents can optimize hardware designs from high-level algorithmic specifications.<n>We introduce an agent factory, a two-stage pipeline that constructs and coordinates multiple autonomous optimization agents.<n>We evaluate the approach on 12 kernels from HLS-Eval and Rodinia-HLS using Claude Code (Opus4.5/4.6) with AMD Vitis HLS.
  arXiv  Detail & Related papers  (2026-03-26T17:57:50Z)
• VeriAgent: A Tool-Integrated Multi-Agent System with Evolving Memory for PPA-Aware RTL Code Generation [52.43796056267479]
  We propose a PPA-aware, tool-integrated multi-agent framework for high-quality verilog code generation.<n>Our framework explicitly incorporates EDA tools into a closed-loop workflow composed of a textitProgrammer Agent, a textitCorrectness Agent, and a textitPPA Agent<n>To support continuous improvement without model retraining, we introduce an textitEvolved Memory Mechanism that externalizes optimization experience into structured memory nodes.
  arXiv  Detail & Related papers  (2026-03-18T11:25:40Z)
• Resource-Efficient Iterative LLM-Based NAS with Feedback Memory [49.44875022114861]
  Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources.<n>We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures.
  arXiv  Detail & Related papers  (2026-03-12T16:00:22Z)
• $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• dInfer: An Efficient Inference Framework for Diffusion Language Models [54.80918957287927]
  Diffusion-based large language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) LLMs.<n>We present dInfer, an efficient and framework for dLLM inference.
  arXiv  Detail & Related papers  (2025-10-09T16:19:42Z)
• Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference [23.633481089469836]
  Auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance.<n>We propose a novel parallel prompt decoding that requires only $0.0002$% trainable parameters, enabling efficient training on a single A100-40GB GPU in just 16 hours.<n>Our approach demonstrates up to 2.49$times$ speedup and maintains a minimal memory overhead of just $0.0004$%.
  arXiv  Detail & Related papers  (2024-05-28T22:19:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.