AgentCompile: An LLM-Guided Compiler for Direct CUDA Inference

• URL: http://arxiv.org/abs/2606.07665v1
• Date: Thu, 04 Jun 2026 03:49:56 GMT
• Title: AgentCompile: An LLM-Guided Compiler for Direct CUDA Inference
• Abstract summary: We present AgentCompile, an LLM-guided inference compiler that uses LLM outputs only as advisory search metadata.<n>Given compiler-derived region summaries and bounded candidate spaces, the LLM proposes semantic labels, candidate priorities, parameter hints, and risk annotations.<n>AgentCompile materializes candidates through graphs templates, checks interface and hardware constraints, validates candidates empirically, selects implementations by measured latency, and falls back when specialization is unsupported or unprofitable.<n>In end-to-end autoregressive generation, AgentCompile averages 5.66x, 4.05x, and 4.26x speedup over PyTorch on Qwen3-1.7B, Qwen
• Score: 58.285369364699
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Transformer inference increasingly depends on specialized compiler and runtime support, but real model graphs still require semantic decisions about which regions are worth specializing and which CUDA implementation families are plausible. We present AgentCompile, an LLM-guided CUDA inference compiler that uses LLM outputs only as advisory search metadata. Given compiler-derived region summaries and bounded candidate spaces, the LLM proposes semantic labels, candidate priorities, parameter hints, and risk annotations; the compiler materializes CUDA candidates through templates, checks interface and hardware constraints, validates candidates empirically, selects implementations by measured latency, and falls back when specialization is unsupported or unprofitable. In end-to-end autoregressive generation, AgentCompile averages 5.66x, 4.05x, and 4.26x speedup over PyTorch eager on Qwen3-1.7B, Qwen3-4B, and Llama-3.2-1B-Instruct, respectively, across five representative workloads. We will open-source the project.

Related papers

• ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation [5.103450268268085]
  We present ParBench, a kernel-centric benchmark framework for evaluating parallel API translation under executable conditions.<n>ParBench fixes the surrounding build, run, and verification infrastructure through declarative benchmark specifications and asks models to translate only kernels.
  arXiv  Detail & Related papers  (2026-06-09T19:00:09Z)
• SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers [55.39407031861402]
  We introduce SCDBench, a dataset and benchmark methodology for smart contract decompilation.<n>The dataset contains 600 real-world Solidity contracts with paired bytecode inputs, ground-truth source code, and replayable semantic checkpoints.<n>We evaluate Claude Opus 4.7, GPT-5.3-Codex, and GLM-5 in a zero-shot decompilation setting, including GLM-5 variants with and without extended reasoning.
  arXiv  Detail & Related papers  (2026-05-27T20:08:47Z)
• ARGUS: Agentic GPU Optimization Guided by Data-Flow Invariants [12.49256588033198]
  LLM-based coding agents can generate functionally correct GPU kernels, yet their performance remains far below hand-optimized libraries on critical computations.<n>We present Argus, an agentic framework that addresses this through data-flow invariants.<n>We evaluate Argus on the AMD MI300X GPU across GEMM, flash attention, and MoE kernels.
  arXiv  Detail & Related papers  (2026-04-16T15:49:31Z)
• A Generalizable Framework for Building Executable Domain-Specific LLMs under Data Scarcity: Demonstration on Semiconductor TCAD Simulation [20.174394305112198]
  We present a framework for building compact, executable domain-specific LLMs in low-resource settings.<n>We demonstrate the framework by instantiating TcadGPT for semiconductor Technology Computer-Aided Design (TCAD)<n>Using 1.5M synthetic QA pairs and an IR-driven DPO dataset, TcadGPT attains 85.6% semantic accuracy and an 80.0% syntax pass rate on SDE executability tests.
  arXiv  Detail & Related papers  (2026-01-15T07:13:34Z)
• nncase: An End-to-End Compiler for Efficient LLM Deployment on Heterogeneous Storage Architectures [7.460240094212613]
  We present nncase, an end-to-end compilation framework designed to unify optimization across diverse targets.<n>nncase integrates three key modules: Auto Vectorize for adapting to heterogeneous computing units, Auto Distribution for searching parallel strategies, and Auto Schedule for maximizing on-chip cache locality.
  arXiv  Detail & Related papers  (2025-12-25T08:27:53Z)
• Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol [83.83217247686402]
  Large Language Models (LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions.<n>Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance.<n>This paper decomposes LLM applications into a three-layer architecture: textbftextitSystem Shell Layer, textbftextitPrompt Orchestration Layer, and textbftextitLLM Inference Core.
  arXiv  Detail & Related papers  (2025-08-28T13:00:28Z)
• SLICEMATE: Accurate and Scalable Static Program Slicing via LLM-Powered Agents [11.069304685402642]
  SliceMate is a novel static program slicing solution powered by Large Language Model (LLM) agents.<n>It bypasses the need for explicit dependency graph construction and achieves superior slicing accuracy.<n>For rigorous evaluation, we construct a new and high-quality benchmark, SliceBench, comprising 2,200 manually annotated Java and Python programs.
  arXiv  Detail & Related papers  (2025-07-25T04:51:47Z)
• CompileAgent: Automated Real-World Repo-Level Compilation with Tool-Integrated LLM-based Agent System [52.048087777953064]
  We propose CompileAgent, an agent framework dedicated to repo-level compilation.<n>CompileAgent integrates five tools and a flow-based agent strategy, enabling interaction with software artifacts for compilation instruction search and error resolution.<n>We show that our method significantly improves the compilation success rate, ranging from 10% to 71%.
  arXiv  Detail & Related papers  (2025-05-07T08:59:14Z)
• ACPO: AI-Enabled Compiler Framework [1.752593459729982]
  This paper presents ACPO: An AI-Enabled Compiler Framework.<n>It provides LLVM with simple and comprehensive tools to benefit from employing ML models for different optimization passes.<n>We show that ACPO can provide a combined speedup of 4.5% on Polybench and 2.4% on Cbench when compared with LLVM's O3.
  arXiv  Detail & Related papers  (2023-12-15T17:49:24Z)
• An LLM Compiler for Parallel Function Calling [68.04566807806071]
  We introduce LLMCompiler, which executes functions in parallel to efficiently orchestrate multiple function calls. We observe consistent latency speedup of up to 3.7x, cost savings of up to 6.7x, and accuracy improvement of up to 9% compared to ReAct.
  arXiv  Detail & Related papers  (2023-12-07T18:32:04Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.