Agentic Harness for Real-World Compilers

• URL: http://arxiv.org/abs/2603.20075v1
• Date: Fri, 20 Mar 2026 15:56:43 GMT
• Title: Agentic Harness for Real-World Compilers
• Abstract summary: We introduce the agentic harness designed to assist large compiler agents in understanding and fixing compiler bugs.<n>Our focus is on LLVM, one of the most widely used compiler infrastructures. Central to-autofix are agent-friendly LLVM tools, and a tailored minimal agent-autofix-mini for fixing LLVM bugs.
• Score: 22.07908620753322
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compilers are critical to modern computing, yet fixing compiler bugs is difficult. While recent large language model (LLM) advancements enable automated bug repair, compiler bugs pose unique challenges due to their complexity, deep cross-domain expertise requirements, and sparse, non-descriptive bug reports, necessitating compiler-specific tools. To bridge the gap, we introduce llvm-autofix, the first agentic harness designed to assist LLM agents in understanding and fixing compiler bugs. Our focus is on LLVM, one of the most widely used compiler infrastructures. Central to llvm-autofix are agent-friendly LLVM tools, a benchmark llvm-bench of reproducible LLVM bugs, and a tailored minimal agent llvm-autofix-mini for fixing LLVM bugs. Our evaluation demonstrates a performance decline of 60% in frontier models when tackling compiler bugs compared with common software bugs. Our minimal agent llvm-autofix-mini also outperforms the state-of-the-art by approximately 22%. This emphasizes the necessity for specialized harnesses like ours to close the loop between LLMs and compiler engineering. We believe this work establishes a foundation for advancing LLM capabilities in complex systems like compilers. GitHub: https://github.com/dtcxzyw/llvm-autofix

Related papers

• Anchored Self-Play for Code Repair [68.0728003844994]
  Code repair is an important capability for language models (LMs)<n>We propose __generator--fixer self-play__, in which a single model is trained with reinforcement learning to generate bugs and fix them.<n>On BugSourceBench, we find that self-play drifts toward difficult but unrealistic bugs.<n>We propose Anchored Self-Play (ASP), which anchors self-play with a small reference set by adding a code-embedding similarity reward for generation and mixing reference bugs into fixer training.
  arXiv  Detail & Related papers  (2026-07-03T17:51:52Z)
• Archer: Towards Agentic Review for Compiler Optimizations [2.4796712345498015]
  We present Archer, the first automated agentic code review tool for compiler optimizations.<n>We evaluate Archer on 70 open PRs and 328 closed PRs in LLVM from the last two months.
  arXiv  Detail & Related papers  (2026-07-02T07:22:40Z)
• LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution [14.546026043461476]
  We introduce LLVM-Bench, the first large-scale benchmark for LLVM issue resolution.<n>We also develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution.<n>Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution.
  arXiv  Detail & Related papers  (2026-07-01T09:50:26Z)
• SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents [3.8743350688734988]
  We study buggy posts and code snippets from Stack Overflow, GitHub, and HuggingFace forums.<n>We introduce AgentDefect, the first benchmark dataset for bugs in LLM agents.<n>We present SelfHeal, a multi-agent system designed to fix bugs in LLM agents.
  arXiv  Detail & Related papers  (2026-04-20T01:28:15Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• Interleaving Large Language Models for Compiler Testing [2.4796712345498015]
  Testing compilers with AI models has shown great promise, but current approaches struggle with two key problems.<n>We propose a novel compiler testing framework that decouples the testing process into two distinct phases.<n>In the online phase, we reuse these code pieces by strategically combining them to build high-quality and valid test programs.
  arXiv  Detail & Related papers  (2025-08-26T11:49:58Z)
• Do AI models help produce verified bug fixes? [62.985237003585674]
  Large Language Models are used to produce corrections to software bugs.<n>This paper investigates how programmers use Large Language Models to complement their own skills.<n>The results are a first step towards a proper role for AI and LLMs in providing guaranteed-correct fixes to program bugs.
  arXiv  Detail & Related papers  (2025-07-21T17:30:16Z)
• CrashFixer: A crash resolution agent for the Linux kernel [58.152358195983155]
  This work builds upon kGym, which shares a benchmark for system-level Linux kernel bugs and a platform to run experiments on the Linux kernel.<n>This paper introduces CrashFixer, the first LLM-based software repair agent that is applicable to Linux kernel bugs.
  arXiv  Detail & Related papers  (2025-04-29T04:18:51Z)
• Finding Missed Code Size Optimizations in Compilers using LLMs [1.90019787465083]
  We develop a novel testing approach which combines large language models with a series of differential testing strategies.<n>Our approach requires fewer than 150 lines of code to implement.<n>To date we have reported 24 confirmed bugs in production compilers.
  arXiv  Detail & Related papers  (2024-12-31T21:47:46Z)
• KGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution [59.20933707301566]
  Large Language Models (LLMs) are consistently improving at increasingly realistic software engineering (SE) tasks. In real-world software stacks, significant SE effort is spent developing foundational system software like the Linux kernel. To evaluate if ML models are useful while developing such large-scale systems-level software, we introduce kGym and kBench.
  arXiv  Detail & Related papers  (2024-07-02T21:44:22Z)
• IRFuzzer: Specialized Fuzzing for LLVM Backend Code Generation [3.4764546186868452]
  We implement IRFuzzer to investigate the effectiveness of specialized fuzzing of the LLVM compiler backend.<n>The mutator in IRFuzzer is capable of generating a wide range of LLVM IR inputs, including structured control flow, vector types, and function definitions.<n>We show that IRFuzzer is more effective than existing fuzzers by fuzzing on 29 mature LLVM backend targets.
  arXiv  Detail & Related papers  (2024-02-07T21:02:33Z)
• DebugBench: Evaluating Debugging Capability of Large Language Models [80.73121177868357]
  DebugBench is a benchmark for Large Language Models (LLMs) It covers four major bug categories and 18 minor types in C++, Java, and Python. We evaluate two commercial and four open-source models in a zero-shot scenario.
  arXiv  Detail & Related papers  (2024-01-09T15:46:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.