What Makes Software Bugs Escape Testing? Evidence from a Large-Scale Empirical Study

• URL: http://arxiv.org/abs/2604.26672v1
• Date: Wed, 29 Apr 2026 13:42:25 GMT
• Title: What Makes Software Bugs Escape Testing? Evidence from a Large-Scale Empirical Study
• Abstract summary: This paper presents a large-scale characterization of pre- and post-release defects across C/C++ and Java systems.<n>Results show that post-release defects are concentrated in older, frequently modified, and high-churn components.
• Score: 2.8358949816586025
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding how software defects manifest and evolve in production environments is critical for improving reliability. While previous research has largely focused on pre-release defects, the nature of residual faults, i.e., those escaping testing and surfacing post-release, remains poorly understood. This paper presents a large-scale characterization of pre- and post-release defects across C/C++ and Java systems, encompassing over 14k defects mined from open-source projects. We employ a broad suite of software metrics to capture diverse code attributes such as complexity, size, structure, and development history. Results show that post-release defects are concentrated in older, frequently modified, and high-churn components, typically requiring longer and more complex fixes than pre-release ones. These findings highlight that residual defects arise more from evolutionary and process dynamics than code structure alone, suggesting that reliability engineering should prioritize targeted testing in mature and complex code regions.

Related papers

• Can Large Language Models Generate Observability-Aware Code? [51.30941152293301]
  We present a systematic study of observability in agent-generated systems.<n>Our results reveal a consistent gap between diagnostic semantics at the source level and fault signals at runtime.<n>We introduce an observability-oriented skill, which can serve as a guidance to improve both diagnostic semantics and fault-signal exposure.
  arXiv  Detail & Related papers  (2026-07-07T03:20:52Z)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
  We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
  arXiv  Detail & Related papers  (2026-03-25T19:26:44Z)
• Characterizing Bugs and Quality Attributes in Quantum Software: A Large-Scale Empirical Study [0.6445605125467574]
  This study presents the first ecosystem-scale longitudinal analysis of software bugs across 123 open source quantum repositories from 2012 to 2024.<n>Full-stack libraries and compilers are the most bug-prone categories due to circuit, gate, and transpilation-related issues.<n>High-severity bugs cluster in cryptography, experimental computing, and compiler toolchains.
  arXiv  Detail & Related papers  (2025-12-31T06:05:49Z)
• The Repeat Offenders: Characterizing and Predicting Extremely Bug-Prone Source Methods [0.8481798330936976]
  ExtremelyBuggy methods constitute only a tiny fraction of all methods, yet frequently account for a disproportionately large share of bugs.<n>These methods are significantly larger, more complex, less readable, and less maintainable than both singly-buggy and non-buggy methods.
  arXiv  Detail & Related papers  (2025-11-27T19:13:24Z)
• Empirical Analysis of Temporal and Spatial Fault Characteristics in Multi-Fault Bug Repositories [45.208325853591475]
  We present an empirical analysis of the temporal and spatial characteristics of faults existing in 16 open-source Java and Python projects.<n>Our findings show that many faults in these software systems are long-lived, leading to the majority of software versions having multiple coexisting faults.
  arXiv  Detail & Related papers  (2025-08-12T11:55:16Z)
• Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points [51.40935517552926]
  We introduce Focused-DPO, a framework that enhances code generation by directing preference optimization towards critical error-prone areas.<n>By focusing on error-prone points, Focused-DPO advances the accuracy and functionality of model-generated code.
  arXiv  Detail & Related papers  (2025-02-17T06:16:02Z)
• LLMs as Continuous Learners: Improving the Reproduction of Defective Code in Software Issues [62.12404317786005]
  EvoCoder is a continuous learning framework for issue code reproduction. Our results show a 20% improvement in issue reproduction rates over existing SOTA methods.
  arXiv  Detail & Related papers  (2024-11-21T08:49:23Z)
• Leveraging Large Language Models for Efficient Failure Analysis in Game Development [47.618236610219554]
  This paper proposes a new approach to automatically identify which change in the code caused a test to fail. The method leverages Large Language Models (LLMs) to associate error messages with the corresponding code changes causing the failure. Our approach reaches an accuracy of 71% in our newly created dataset, which comprises issues reported by developers at EA over a period of one year.
  arXiv  Detail & Related papers  (2024-06-11T09:21:50Z)
• Code Revert Prediction with Graph Neural Networks: A Case Study at J.P. Morgan Chase [10.961209762486684]
  Code revert prediction aims to forecast or predict the likelihood of code changes being reverted or rolled back in software development. Previous methods for code defect detection relied on independent features but ignored relationships between code scripts. This paper presents a systematic empirical study for code revert prediction that integrates the code import graph with code features.
  arXiv  Detail & Related papers  (2024-03-14T15:54:29Z)
• Patch2QL: Discover Cognate Defects in Open Source Software Supply Chain With Auto-generated Static Analysis Rules [1.9591497166224197]
  We propose a novel technique for detecting cognate defects in OSS through the automatic generation of SAST rules. Specifically, it extracts key syntax and semantic information from pre- and post-patch versions of code. We have implemented a prototype tool called Patch2QL and applied it to fundamental OSS in C/C++.
  arXiv  Detail & Related papers  (2024-01-23T02:23:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.