Improving Spectrum-Based Localization of Multiple Faults by Iterative Test Suite Reduction

• URL: http://arxiv.org/abs/2306.09892v1
• Date: Fri, 16 Jun 2023 15:00:40 GMT
• Title: Improving Spectrum-Based Localization of Multiple Faults by Iterative Test Suite Reduction
• Authors: Dylan Callaghan, Bernd Fischer
• Abstract summary: We present FLITSR, a novel SBFL extension that improves the localization of a given base metric in the presence of multiple faults. For all three spectrum types we consistently see substantial reductions of the average wasted efforts at different fault levels, of 30%-90% over the best base metric. For the method-level real faults, FLITSR also substantially outperforms GRACE, a state-of-the-art learning-based fault localizer.
• Score: 0.30458514384586394
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spectrum-based fault localization (SBFL) works well for single-fault programs but its accuracy decays for increasing fault numbers. We present FLITSR (Fault Localization by Iterative Test Suite Reduction), a novel SBFL extension that improves the localization of a given base metric specifically in the presence of multiple faults. FLITSR iteratively selects reduced versions of the test suite that better localize the individual faults in the system. This allows it to identify and re-rank faults ranked too low by the base metric because they were masked by other program elements. We evaluated FLITSR over method-level spectra from an existing large synthetic dataset comprising 75000 variants of 15 open-source projects with up to 32 injected faults, as well as method-level and statement-level spectra from a new dataset with 326 true multi-fault versions from the Defects4J benchmark set containing up to 14 real faults. For all three spectrum types we consistently see substantial reductions of the average wasted efforts at different fault levels, of 30%-90% over the best base metric, and generally similarly large increases in precision and recall, albeit with larger variance across the underlying projects. For the method-level real faults, FLITSR also substantially outperforms GRACE, a state-of-the-art learning-based fault localizer.

Related papers

• Enhancing Fault Localization Through Ordered Code Analysis with LLM Agents and Self-Reflection [8.22737389683156]
  Large Language Models (LLMs) offer promising improvements in fault localization by enhancing code comprehension and reasoning. We introduce LLM4FL, a novel LLM-agent-based fault localization approach that integrates SBFL rankings with a divide-and-conquer strategy. Our results demonstrate that LLM4FL outperforms AutoFL by 19.27% in Top-1 accuracy and surpasses state-of-the-art supervised techniques such as DeepFL and Grace.
  arXiv  Detail & Related papers  (2024-09-20T16:47:34Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [80.18342600996601]
  Large language models (LLMs) produce code that is shorter yet more complicated as compared to canonical solutions. We develop a taxonomy of bugs for incorrect codes that includes three categories and 12 sub-categories, and analyze the root cause for common bug types. We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• Leveraging Stack Traces for Spectrum-based Fault Localization in the Absence of Failing Tests [44.13331329339185]
  We introduce a new approach, SBEST, that integrates stack trace data with test coverage to enhance fault localization. Our approach shows a significant improvement, increasing Mean Average Precision (MAP) by 32.22% and Mean Reciprocal Rank (MRR) by 17.43% over traditional stack trace ranking methods.
  arXiv  Detail & Related papers  (2024-05-01T15:15:52Z)
• Demystifying Faulty Code with LLM: Step-by-Step Reasoning for Explainable Fault Localization [5.7821087202452]
  This study investigates the step-by-step reasoning for explainable fault localization. We created a dataset of faulty code files, along with explanations for 600 faulty lines. We found that for 22 out of the 30 randomly sampled cases, FuseFL generated correct explanations.
  arXiv  Detail & Related papers  (2024-03-15T17:47:20Z)
• EBFT: Effective and Block-Wise Fine-Tuning for Sparse LLMs [68.41135269685576]
  Existing methods for fine-tuning sparse LLMs often suffer from resource-intensive requirements and high retraining costs. We propose an efficient and fast framework for fine-tuning sparse LLMs based on minimizing reconstruction error. Our approach involves sampling a small dataset for calibration and utilizing backpropagation to iteratively optimize block-wise reconstruction error.
  arXiv  Detail & Related papers  (2024-02-19T09:55:32Z)
• Large Language Models for Test-Free Fault Localization [11.080712737595174]
  We propose a language model based fault localization approach that locates buggy lines of code without any test coverage information. We fine-tune language models with 350 million, 6 billion, and 16 billion parameters on small, manually curated corpora of buggy programs. Our empirical evaluation shows that LLMAO improves the Top-1 results over the state-of-the-art machine learning fault localization (MLFL) baselines by 2.3%-54.4%, and Top-5 results by 14.4%-35.6%.
  arXiv  Detail & Related papers  (2023-10-03T01:26:39Z)
• Debugging Flaky Tests using Spectrum-based Fault Localization [14.609208863749831]
  Flaky tests hamper regression testing as they destroy trust and waste computational and human resources. We introduce SFFL (Spectrum-based Flaky Fault localization), an extension of traditional coverage-based SFL. An evaluation on 101 flaky tests taken from 48 open-source Python projects demonstrates that SFFL is effective.
  arXiv  Detail & Related papers  (2023-05-08T14:40:05Z)
• Rethinking Normalization Methods in Federated Learning [92.25845185724424]
  Federated learning (FL) is a popular distributed learning framework that can reduce privacy risks by not explicitly sharing private data. We show that external covariate shifts will lead to the obliteration of some devices' contributions to the global model.
  arXiv  Detail & Related papers  (2022-10-07T01:32:24Z)
• Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels [92.98756432746482]
  We study a weakly supervised problem called learning with complementary labels. We show that the quality of gradient estimation matters more in risk minimization. We propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance.
  arXiv  Detail & Related papers  (2020-07-05T04:19:37Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.