Impact of Large Language Models of Code on Fault Localization

• URL: http://arxiv.org/abs/2408.09657v1
• Date: Mon, 19 Aug 2024 02:36:07 GMT
• Title: Impact of Large Language Models of Code on Fault Localization
• Authors: Suhwan Ji, Sanghwa Lee, Changsup Lee, Hyeonseung Im, Yo-Sub Han,
• Abstract summary: We propose a simple but effective sequence generation approach for fine-tuning large language models of code for FL tasks. Specifically, we fine-tune representative encoder, encoder-decoder, and decoder-based 13 LLMCs for FL tasks. Experimental results show that LLMCs fine-tuned with our approach successfully pinpoint error positions in 50.6%, 64.2%, and 72.3% of 1,291 methods in Defects4J for Top-2/3/5 prediction.
• Score: 2.936007114555107
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Identifying the point of error is imperative in software debugging. Traditional fault localization (FL) techniques rely on executing the program and using the code coverage matrix in tandem with test case results to calculate a suspiciousness score for each function or line. Recently, learning-based FL techniques have harnessed machine learning models to extract meaningful features from the code coverage matrix and improve FL performance. These techniques, however, require compilable source code, existing test cases, and specialized tools for generating the code coverage matrix for each programming language of interest. In this paper, we propose, for the first time, a simple but effective sequence generation approach for fine-tuning large language models of code (LLMCs) for FL tasks. LLMCs have recently received much attention for various software engineering problems. In line with these, we leverage the innate understanding of code that LLMCs have acquired through pre-training on large code corpora. Specifically, we fine-tune representative encoder, encoder-decoder, and decoder-based 13 LLMCs for FL tasks. Unlike previous approaches, LLMCs can analyze code sequences even with syntactic errors, since they do not rely on compiled input. Still, they have a limitation on the length of the input data. Therefore, for a fair comparison with existing FL techniques, we extract methods with errors from the project-level benchmark, Defects4J, and analyze them at the line level. Experimental results show that LLMCs fine-tuned with our approach successfully pinpoint error positions in 50.6\%, 64.2\%, and 72.3\% of 1,291 methods in Defects4J for Top-1/3/5 prediction, outperforming the best learning-based state-of-the-art technique by up to 1.35, 1.12, and 1.08 times, respectively. Our findings suggest promising research directions for FL and automated program repair tasks using LLMCs.

Related papers

• Utilizing Precise and Complete Code Context to Guide LLM in Automatic False Positive Mitigation [3.0538467265507574]
  Application Security Testing(SAST) tools are crucial for early bug detection and code quality but often generate false positives that slow development. Large Language Models, adept at understanding natural language and code, offer promising ways to improve the accuracy and usability of SAST tools. Our work emphasizes the critical impact of precise and complete code context and highlights the potential of combining program analysis with LLMs.
  arXiv  Detail & Related papers  (2024-11-05T13:24:56Z)
• Enhancing Fault Localization Through Ordered Code Analysis with LLM Agents and Self-Reflection [8.22737389683156]
  Large Language Models (LLMs) offer promising improvements in fault localization by enhancing code comprehension and reasoning. We introduce LLM4FL, a novel LLM-agent-based fault localization approach that integrates SBFL rankings with a divide-and-conquer strategy. Our results demonstrate that LLM4FL outperforms AutoFL by 19.27% in Top-1 accuracy and surpasses state-of-the-art supervised techniques such as DeepFL and Grace.
  arXiv  Detail & Related papers  (2024-09-20T16:47:34Z)
• Program Slicing in the Era of Large Language Models [7.990456190723922]
  Program slicing is a critical technique in software engineering, enabling developers to isolate relevant portions of code. This study investigates the application of large language models (LLMs) to both static and dynamic program slicing.
  arXiv  Detail & Related papers  (2024-09-19T00:07:56Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [80.18342600996601]
  Large language models (LLMs) produce code that is shorter yet more complicated as compared to canonical solutions. We develop a taxonomy of bugs for incorrect codes that includes three categories and 12 sub-categories, and analyze the root cause for common bug types. We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback [58.20547418182074]
  We introduce StepCoder, a novel framework for code generation, consisting of two main components. CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks. FGO only optimize the model by masking the unexecuted code segments to provide Fine-Grained Optimization. Our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks.
  arXiv  Detail & Related papers  (2024-02-02T13:14:31Z)
• SparseCoder: Advancing Source Code Analysis with Sparse Attention and Learned Token Pruning [10.067863549963834]
  This paper introduces SparseCoder, an innovative approach incorporating sparse attention and learned token pruning. Compared to previous state-of-the-art models CodeBERT, RoBERTa, and CodeT5, our experiments demonstrate that SparseCoder can handle significantly longer input sequences. SparseCoder is four times faster than other methods measured in, achieving a 50% reduction in floating point operations per second.
  arXiv  Detail & Related papers  (2023-10-11T01:11:30Z)
• Zero-Shot Detection of Machine-Generated Codes [83.0342513054389]
  This work proposes a training-free approach for the detection of LLMs-generated codes. We find that existing training-based or zero-shot text detectors are ineffective in detecting code. Our method exhibits robustness against revision attacks and generalizes well to Java codes.
  arXiv  Detail & Related papers  (2023-10-08T10:08:21Z)
• Large Language Models for Test-Free Fault Localization [11.080712737595174]
  We propose a language model based fault localization approach that locates buggy lines of code without any test coverage information. We fine-tune language models with 350 million, 6 billion, and 16 billion parameters on small, manually curated corpora of buggy programs. Our empirical evaluation shows that LLMAO improves the Top-1 results over the state-of-the-art machine learning fault localization (MLFL) baselines by 2.3%-54.4%, and Top-5 results by 14.4%-35.6%.
  arXiv  Detail & Related papers  (2023-10-03T01:26:39Z)
• CodeT5+: Open Code Large Language Models for Code Understanding and Generation [72.1638273937025]
  Large language models (LLMs) pretrained on vast source code have achieved prominent progress in code intelligence. CodeT5+ is a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks. We extensively evaluate CodeT5+ on over 20 code-related benchmarks in different settings, including zero-shot, finetuning, and instruction-tuning.
  arXiv  Detail & Related papers  (2023-05-13T14:23:07Z)
• Machine Learning-Aided Efficient Decoding of Reed-Muller Subcodes [59.55193427277134]
  Reed-Muller (RM) codes achieve the capacity of general binary-input memoryless symmetric channels. RM codes only admit limited sets of rates. Efficient decoders are available for RM codes at finite lengths.
  arXiv  Detail & Related papers  (2023-01-16T04:11:14Z)
• Fault-Aware Neural Code Rankers [64.41888054066861]
  We propose fault-aware neural code rankers that can predict the correctness of a sampled program without executing it. Our fault-aware rankers can significantly increase the pass@1 accuracy of various code generation models.
  arXiv  Detail & Related papers  (2022-06-04T22:01:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.