Similar Pattern Annotation via Retrieval Knowledge for LLM-Based Test Code Fault Localization

• URL: http://arxiv.org/abs/2605.07957v1
• Date: Fri, 08 May 2026 16:20:58 GMT
• Title: Similar Pattern Annotation via Retrieval Knowledge for LLM-Based Test Code Fault Localization
• Abstract summary: Failures in software can originate from faulty system test scripts.<n>Test Code Fault localization (TCFL) has received significantly less attention despite its importance in continuous integration environments.<n>We propose SPARK, a framework that integrates accumulated debug knowledge fromCI environments into Large Language Model (LLM)-based TCFL.
• Score: 5.9162004338597365
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software failures remain a major challenge in modern software development, and identifying the code elements responsible for failures is a time-consuming debugging task. While extensive research has focused on fault localization in the system under test (SUT), failures can also originate from faulty system test scripts. This problem, known as Test Code Fault Localization (TCFL), has received significantly less attention despite its importance in continuous integration (CI) environments where large test suites are executed frequently. TCFL is particularly challenging because it typically operates under black-box conditions, relies on limited diagnostic signals such as error messages and partial logs, and involves large system-level test scripts that expand the fault localization search space. In this paper, we propose SPARK, a framework that integrates accumulated debugging knowledge from continuous integration (CI) environments into Large Language Model (LLM)-based TCFL. Given a newly observed failing test case, SPARK retrieves similar fault-labeled test cases from a debugging knowledge corpus and selectively annotates suspicious lines of the failing test based on their similarity to previously observed fault patterns. These annotations guide the LLM's reasoning while maintaining scalability and avoiding the prompt-length explosion common to naive retrieval-augmented approaches. We evaluate SPARK on three industrial datasets containing real-world faulty Python test cases from different software products. The results show that SPARK consistently improves fault localization effectiveness compared to the existing LLM-based TCFL baseline while maintaining comparable inference cost and token usage. In particular, the approach advances the state of the art by identifying more correct faulty locations in complex test cases containing multiple faults.

Related papers

• DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems [61.11435469308521]
  We propose DCFA, a training-free framework for failure attribution in large language model systems.<n> DCFA integrates a global module that constructs structured causal-inspired dependency graphs from system traces.<n>Experiments show that DCFA improves step-level accuracy by up to 8.27% over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-09-04T05:29:31Z)
• Hierarchical Fault Localization for Autonomous Driving Systems with Hypothesis Validation and Intent Analysis [14.387618464803774]
  We present HINT, a two-phase framework for hierarchical ADS fault localization based on hypothesis validation and intent analysis.<n>In Phase I, HINT transforms failure-triggering execution recordings into multi-modal abstractions.<n>In Phase II, it reconstructs design-side intent and implementation-side behavior, then localizes suspicious code through reliability-aware consistency checking.
  arXiv  Detail & Related papers  (2026-07-14T10:17:01Z)
• How do Execution Features Improve Statistical Fault Localization? An Empirical Study [5.85833888070275]
  This study evaluates whether augmenting Statistical fault localization with execution features improves localization accuracy and developer-oriented inspection effort.<n>We extract execution features with EFDD for all Tests4Py subjects, train per-subject random forests, map importances to source lines, and combine the resulting weights with established SFL formulas.
  arXiv  Detail & Related papers  (2026-06-29T14:07:47Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• How LLMs Fail and Generalize in RTL Coding for Hardware Design? [56.361436215029045]
  We introduce a new error taxonomy grounded in problem solvability, inspired by cognitive theory.<n>Our taxonomy categorizes failures into syntactic, semantic, solvable functional, and unsolvable functional types.
  arXiv  Detail & Related papers  (2026-04-26T14:34:49Z)
• ProbeLLM: Automating Principled Diagnosis of LLM Failures [89.44131968886184]
  We propose ProbeLLM, a benchmark-agnostic automated probing framework that elevates weakness discovery from individual failures to structured failure modes.<n>By restricting probing to verifiable test cases and leveraging tool-augmented generation and verification, ProbeLLM grounds failure discovery in reliable evidence.
  arXiv  Detail & Related papers  (2026-02-13T14:33:13Z)
• Adapting Language Balance in Code-Switching Speech [60.296574524609575]
  Large foundational models still struggle against code-switching test cases.<n>We use differentiable surrogates to mitigate context bias during generation.<n>Experiments with Arabic and Chinese-English showed that the models are able to predict the switching places more correctly.
  arXiv  Detail & Related papers  (2025-10-21T15:23:55Z)
• When the Code Autopilot Breaks: Why LLMs Falter in Embedded Machine Learning [0.8880611506199766]
  We show how prompt format, model behavior, and structural assumptions influence both success rates and failure characteristics.<n>Our analysis reveals a diverse set of error-prone behaviors, including format-induced misinterpretations and runtime-disruptive code that compiles but breaks downstream.
  arXiv  Detail & Related papers  (2025-09-13T19:00:04Z)
• Black-Box Test Code Fault Localization Driven by Large Language Models and Execution Estimation [7.040370156228408]
  We introduce a fully static, LLM-driven approach for system test code fault localization.<n>Our method uses a single failure execution log to estimate the test's execution trace.<n>We evaluate our technique at function, block, and line levels using an industrial dataset of faulty test cases.
  arXiv  Detail & Related papers  (2025-06-23T19:04:51Z)
• CFaults: Model-Based Diagnosis for Fault Localization in C Programs with Multiple Test Cases [2.8853600644505266]
  This paper introduces a novel fault localization approach for C programs with multiple faults.<n>CFaults leverages Model-Based Diagnosis (MBD) with multiple observations and aggregates all failing test cases into a unified MaxSAT formula.<n> Experimental results on two benchmark sets of C programs, TCAS and C-Pack-IPAs, show that CFaults is faster than other FBFL approaches.
  arXiv  Detail & Related papers  (2024-07-12T15:14:49Z)
• Large-scale, Independent and Comprehensive study of the power of LLMs for test case generation [11.517293765116307]
  Unit testing is essential for software reliability, yet manual test creation is time-consuming and often neglected.<n>This study presents the first large-scale empirical evaluation of LLM-generated unit tests at the class level.
  arXiv  Detail & Related papers  (2024-06-28T20:38:41Z)
• FactCHD: Benchmarking Fact-Conflicting Hallucination Detection [64.4610684475899]
  FactCHD is a benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. We introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2.
  arXiv  Detail & Related papers  (2023-10-18T16:27:49Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• Shortcomings of Question Answering Based Factuality Frameworks for Error Localization [51.01957350348377]
  We show that question answering (QA)-based factuality metrics fail to correctly identify error spans in generated summaries. Our analysis reveals a major reason for such poor localization: questions generated by the QG module often inherit errors from non-factual summaries which are then propagated further into downstream modules. Our experiments conclusively show that there exist fundamental issues with localization using the QA framework which cannot be fixed solely by stronger QA and QG models.
  arXiv  Detail & Related papers  (2022-10-13T05:23:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.