Finding Missed Code Size Optimizations in Compilers using LLMs

• URL: http://arxiv.org/abs/2501.00655v1
• Date: Tue, 31 Dec 2024 21:47:46 GMT
• Title: Finding Missed Code Size Optimizations in Compilers using LLMs
• Authors: Davide Italiano, Chris Cummins,
• Abstract summary: We develop a novel testing approach which combines large language models with a series of differential testing strategies. Our approach requires fewer than 150 lines of code to implement. To date we have reported 24 confirmed bugs in production compilers.
• Score: 1.90019787465083
• License:
• Abstract: Compilers are complex, and significant effort has been expended on testing them. Techniques such as random program generation and differential testing have proved highly effective and have uncovered thousands of bugs in production compilers. The majority of effort has been expended on validating that a compiler produces correct code for a given input, while less attention has been paid to ensuring that the compiler produces performant code. In this work we adapt differential testing to the task of identifying missed optimization opportunities in compilers. We develop a novel testing approach which combines large language models (LLMs) with a series of differential testing strategies and use them to find missing code size optimizations in C / C++ compilers. The advantage of our approach is its simplicity. We offload the complex task of generating random code to an off-the-shelf LLM, and use heuristics and analyses to identify anomalous compiler behavior. Our approach requires fewer than 150 lines of code to implement. This simplicity makes it extensible. By simply changing the target compiler and initial LLM prompt we port the approach from C / C++ to Rust and Swift, finding bugs in both. To date we have reported 24 confirmed bugs in production compilers, and conclude that LLM-assisted testing is a promising avenue for detecting optimization bugs in real world compilers.

Related papers

• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [80.18342600996601]
  Large language models (LLMs) produce code that is shorter yet more complicated as compared to canonical solutions. We develop a taxonomy of bugs for incorrect codes that includes three categories and 12 sub-categories, and analyze the root cause for common bug types. We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback [58.20547418182074]
  We introduce StepCoder, a novel framework for code generation, consisting of two main components. CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks. FGO only optimize the model by masking the unexecuted code segments to provide Fine-Grained Optimization. Our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks.
  arXiv  Detail & Related papers  (2024-02-02T13:14:31Z)
• Evolutionary Generative Fuzzing for Differential Testing of the Kotlin Compiler [14.259471945857431]
  We investigate the effectiveness of differential testing in finding bugs within the Kotlin compilers developed at JetBrains. We propose a black-box generative approach that creates input programs for the K1 and K2 compilers. Our case study shows that the proposed approach effectively detects bugs in K1 and K2; these bugs have been confirmed and (some) fixed by JetBrains developers.
  arXiv  Detail & Related papers  (2024-01-12T16:01:12Z)
• Chain of Code: Reasoning with a Language Model-Augmented Code Emulator [115.16975276693267]
  We propose Chain of Code, a simple yet surprisingly effective extension that improves LM code-driven reasoning. The key idea is to encourage LMs to format semantic sub-tasks in a program as flexible pseudocode that the interpreter can explicitly catch.
  arXiv  Detail & Related papers  (2023-12-07T17:51:43Z)
• WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models [11.33856613057612]
  We propose WhiteFox, the first white-box compiler fuzzer using Large Language Models with source-code information. WhiteFox can generate high-quality test programs to exercise deep optimizations, practicing up to 8X more than state-of-the-art fuzzers. WhiteFox has found 101 bugs for the DL compilers, with 92 confirmed as previously unknown and 70 fixed.
  arXiv  Detail & Related papers  (2023-10-24T16:39:06Z)
• A Survey of Modern Compiler Fuzzing [0.0]
  This survey provides a summary of the research efforts for understanding and addressing compilers defects. It covers researchers investigation and expertise on compilers bugs, such as their symptoms and root causes. In addition, it covers researchers efforts in designing fuzzing techniques, including constructing test programs and designing test oracles.
  arXiv  Detail & Related papers  (2023-06-12T06:03:51Z)
• A Static Evaluation of Code Completion by Large Language Models [65.18008807383816]
  Execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. static analysis tools such as linters, which can detect errors without running the program, haven't been well explored for evaluating code generation models. We propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees.
  arXiv  Detail & Related papers  (2023-06-05T19:23:34Z)
• LEVER: Learning to Verify Language-to-Code Generation with Execution [64.36459105535]
  We propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results. LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci) and achieves new state-of-the-art results on all of them.
  arXiv  Detail & Related papers  (2023-02-16T18:23:22Z)
• Interactive Code Generation via Test-Driven User-Intent Formalization [60.90035204567797]
  Large language models (LLMs) produce code from informal natural language (NL) intent. It is hard to define a notion of correctness since natural language can be ambiguous and lacks a formal semantics. We describe a language-agnostic abstract algorithm and a concrete implementation TiCoder.
  arXiv  Detail & Related papers  (2022-08-11T17:41:08Z)
• CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research [26.06438868492976]
  Interest in applying Artificial Intelligence (AI) techniques to compiler optimizations is increasing rapidly. But compiler research has a high entry barrier. We introduce CompilerGym, a set of environments for real world compiler optimization tasks. We also introduce a toolkit for exposing new optimization tasks to compiler researchers.
  arXiv  Detail & Related papers  (2021-09-17T01:02:27Z)
• Configuring Test Generators using Bug Reports: A Case Study of GCC Compiler and Csmith [2.1016374925364616]
  This paper uses the code snippets in the bug reports to guide the test generation. We evaluate this approach on eight versions of GCC. We find that our approach provides higher coverage and triggers more miscompilation failures than the state-of-the-art test generation techniques for GCC.
  arXiv  Detail & Related papers  (2020-12-19T11:25:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.