Can Coding Agents Implement Missed Compiler Optimizations? Evaluating LLM Agents on LLVM Peephole Optimizations

• URL: http://arxiv.org/abs/2607.02684v1
• Date: Thu, 02 Jul 2026 18:25:29 GMT
• Title: Can Coding Agents Implement Missed Compiler Optimizations? Evaluating LLM Agents on LLVM Peephole Optimizations
• Abstract summary: We introduce PeepholeBench, an evaluation framework whose tasks are constructed from real-world missed peephole optimizations.<n>We benchmark state-of-the-art coding agents on fixing missed peephole optimizations in LLVM's InstCombine pass.<n>We observe a tension between correctness and profitability, and no agent matches human developers on both dimensions at once.
• Score: 3.2479630553628156
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Coding agents built on large language models are now capable of patching sizable real-world codebases, yet whether they can develop compiler optimizations remains an open question. To study this question, we introduce PeepholeBench, an evaluation framework whose tasks are constructed from real-world missed peephole optimizations reported against LLVM's InstCombine pass. Since missed peephole optimizations are typically fixed with small, localized patches, they offer a well-scoped but demanding testbed for coding agents: a correct fix demands rigorous reasoning about program semantics along with familiarity with optimizer-specific conventions. PeepholeBench derives its tasks from 21 resolved LLVM issues and 19 merged pull requests (PRs), supplies agents with only the issue context that existed before each fix, and assesses the resulting patches for both correctness and profitability. With PeepholeBench, we benchmark state-of-the-art coding agents on fixing missed peephole optimizations in LLVM's InstCombine pass, measuring their patches against the corresponding human-written fixes. We observe a tension between correctness and profitability, and no agent matches human developers on both dimensions at once. The dominant failure modes are overly narrow transformations and misuse of LLVM-specific mechanisms, errors that existing test suites rarely expose. Together, these results establish PeepholeBench as a realistic and challenging benchmark for coding agents, and suggest future directions for building agents that can more dependably assist compiler optimization development.

Related papers

• The Optimizer Is the Agent: Reasoning-Driven Search across Prompts, Programs, and ML Workflows [72.9051182568928]
  We present ReASearch, a unified framework for reasoning-driven optimization.<n>With a shared agent loop and domain-specific tools, ReASearch instantiates the exact same scaffold to optimize prompts, programs, and ML.<n>We observe that complex search behaviors emerge naturally from the agent's reasoning process.
  arXiv  Detail & Related papers  (2026-08-07T02:15:39Z)
• Understanding Agent-Based Patching of Compiler Missed Optimizations [22.677231886742547]
  We identify a significant challenge that patching a missed optimization requires more than just fixing the reported case.<n>We construct a benchmark of real-world LLVM missed optimization issues and compare agent-generated patches with patches from developers in terms of optimization scope.
  arXiv  Detail & Related papers  (2026-07-02T16:12:32Z)
• Archer: Towards Agentic Review for Compiler Optimizations [2.4796712345498015]
  We present Archer, the first automated agentic code review tool for compiler optimizations.<n>We evaluate Archer on 70 open PRs and 328 closed PRs in LLVM from the last two months.
  arXiv  Detail & Related papers  (2026-07-02T07:22:40Z)
• AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation [59.964989458924585]
  AdaExplore is an agent framework that enables self-improvement via accumulated execution feedback for performance-critical kernel code generation.<n>In the adaptation stage, the agent synthesizes tasks and converts recurring failures into a reusable memory of validity rules.<n>In the search stage, the agent organizes candidate kernels as a tree and alternates between small local refinements and larger structural regeneration.
  arXiv  Detail & Related papers  (2026-04-17T18:25:03Z)
• Execution-Verified Reinforcement Learning for Optimization Modeling [49.171122807323634]
  Execution-verified learning framework treats a mathematical programming solver as a deterministic, interactive verifier.<n>Experiments on NL4OPT, MAMO, IndustryOR, and OptiBench across Gurobi, OR-Tools, and COPT show that EVOM matches or outperforms process-supervised SFT.
  arXiv  Detail & Related papers  (2026-04-01T03:39:11Z)
• ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads? [0.8749675983608171]
  We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference tasks.<n>We curated 54 tasks from merged pull requests with measurable performance improvements.
  arXiv  Detail & Related papers  (2026-02-23T08:37:53Z)
• REFINE: Enhancing Program Repair Agents through Context-Aware Patch Refinement [12.995571513415905]
  Large Language Models (LLMs) have recently shown strong potential in automatic program repair (APR)<n>LLMs often struggle to produce correct fixes due to limited understanding of code context and over-reliance on incomplete test suites.<n>We propose a novel patch refinement framework, Refine, that systematically transforms Draft Patches into correct ones.
  arXiv  Detail & Related papers  (2025-10-04T00:34:32Z)
• Leveraging Large Language Models to Detect Missed Peephole Optimizations [7.48961433936748]
  peephole optimization is a critical class of compiler optimizations.<n>Previous methods either do not scale well or can only capture a limited subset of peephole optimizations.<n>We propose Lampo, a novel framework that combines the creative but unreliable code optimization ability of LLMs with rigorous correctness verification.
  arXiv  Detail & Related papers  (2025-08-22T06:36:42Z)
• Do AI models help produce verified bug fixes? [62.985237003585674]
  Large Language Models are used to produce corrections to software bugs.<n>This paper investigates how programmers use Large Language Models to complement their own skills.<n>The results are a first step towards a proper role for AI and LLMs in providing guaranteed-correct fixes to program bugs.
  arXiv  Detail & Related papers  (2025-07-21T17:30:16Z)
• SolBench: A Dataset and Benchmark for Evaluating Functional Correctness in Solidity Code Completion and Repair [51.0686873716938]
  We introduce SolBench, a benchmark for evaluating the functional correctness of Solidity smart contracts generated by code completion models.<n>We propose a Retrieval-Augmented Code Repair framework to verify functional correctness of smart contracts.<n>Results show that code repair and retrieval techniques effectively enhance the correctness of smart contract completion while reducing computational costs.
  arXiv  Detail & Related papers  (2025-03-03T01:55:20Z)
• RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair [75.40584530380589]
  We propose a novel Retrieval-Augmented Patch Generation framework (RAP-Gen) RAP-Gen explicitly leveraging relevant fix patterns retrieved from a list of previous bug-fix pairs. We evaluate RAP-Gen on three benchmarks in two programming languages, including the TFix benchmark in JavaScript, and Code Refinement and Defects4J benchmarks in Java.
  arXiv  Detail & Related papers  (2023-09-12T08:52:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.