Real-World Fault Detection for C-Extended Python Projects with Automated Unit Test Generation

• URL: http://arxiv.org/abs/2603.06107v1
• Date: Fri, 06 Mar 2026 10:05:29 GMT
• Title: Real-World Fault Detection for C-Extended Python Projects with Automated Unit Test Generation
• Abstract summary: Python libraries use C-extensions for performance-critical operations.<n>Exceptions raised in C can bypass Python's exception handling and cause the entire interpreter to crash.<n>We propose separating the generation and execution stages of the test-generation process.
• Score: 14.579734219257224
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many popular Python libraries use C-extensions for performance-critical operations allowing users to combine the best of the two worlds: The simplicity and versatility of Python and the performance of C. A drawback of this approach is that exceptions raised in C can bypass Python's exception handling and cause the entire interpreter to crash. These crashes are real faults if they occur when calling a public API. While automated test generation should, in principle, detect such faults, crashes in native code can halt the test process entirely, preventing detection or reproduction of the underlying errors and inhibiting coverage of non-crashing parts of the code. To overcome this problem, we propose separating the generation and execution stages of the test-generation process. We therefore adapt Pynguin, an automated test case generation tool for Python, to use subprocess-execution. Executing each generated test in an isolated subprocess prevents a crash from halting the test generation process itself. This allows us to (1) detect such faults, (2) generate reproducible crash-revealing test cases for them, (3) allow studying the underlying faults, and (4) enable test generation for non-crashing parts of the code. To evaluate our approach, we created a dataset consisting of 1648 modules from 21 popular Python libraries with C-extensions. Subprocess-execution allowed automated testing of up to 56.5% more modules and discovered 213 unique crash causes, revealing 32 previously unknown faults.

Related papers

• Beyond Source: An Empirical Study of Python Bytecode Security Risks [9.295635962179816]
  Python runtimes can execute bytecode directly through.pyc files, compiled-only modules, and marshalled code objects.<n>Across 1,034,843 collected PyPI artifacts, we identify 7,388 bytecode-containing artifacts, including 228,578.pyc files and 28,193 artifact-local source-less.pyc files.
  arXiv  Detail & Related papers  (2026-08-13T05:54:24Z)
• ProgramBench: Can Language Models Rebuild Programs From Scratch? [59.40748183470308]
  ProgramBench measures the ability of software engineering agents to develop software holisitically.<n>End-to-end behavioral tests are generated via agent-driven fuzzing.<n>Models favor monolithic, single-file implementations that diverge sharply from human-written code.
  arXiv  Detail & Related papers  (2026-05-05T09:17:02Z)
• Constraint-Guided Unit Test Generation for Machine Learning Libraries [8.883254370291256]
  Machine learning (ML) libraries such as PyTorch and tensors are essential for a wide range of modern applications.<n> Ensuring the correctness of ML libraries through testing is crucial.<n>In this paper, we present PynguinML, an approach that improves the Pynguin test generator to leverage these constraints.
  arXiv  Detail & Related papers  (2025-10-10T08:02:15Z)
• Reflective Unit Test Generation for Precise Type Error Detection with Large Language Models [13.969152395348653]
  RTED is a type-aware test generation technique for automatically detecting Python type errors.<n>We show that RTED can detect 22-29 more benchmarked type errors than four state-of-the-art techniques.<n>It is also capable of producing fewer false positives, achieving an improvement of 173.9%-245.9% in precision.
  arXiv  Detail & Related papers  (2025-07-03T05:10:33Z)
• Bugs in the Shadows: Static Detection of Faulty Python Refactorings [44.115219601924856]
  Python's dynamic type system poses significant challenges for automated code transformations.<n>Our analysis uncovered 29 bugs across four types from a total of 1,152 attempts.<n>These results highlight the need to improve the robustness of current Python tools to ensure the correctness of automated code transformations.
  arXiv  Detail & Related papers  (2025-07-01T18:03:56Z)
• PyResBugs: A Dataset of Residual Python Bugs for Natural Language-Driven Fault Injection [5.383910843560784]
  PyResBugs is a curated dataset of residual bugs from major Python frameworks.<n>Each bug is paired with its corresponding fault-free (fixed) version and annotated with multi-level natural language (NL) descriptions.
  arXiv  Detail & Related papers  (2025-05-09T04:39:09Z)
• Type-aware LLM-based Regression Test Generation for Python Programs [13.631541369653066]
  Test4Py is a novel framework that enhances type correctness in automated test generation for Python.<n>Test4Py integrates an iterative repair procedure that progressively refines generated test cases to improve coverage.<n>In an evaluation on 183 real-world Python modules, Test4Py achieved an average statement coverage of 83.0% and branch coverage of 70.8%.
  arXiv  Detail & Related papers  (2025-03-18T08:07:17Z)
• PyPulse: A Python Library for Biosignal Imputation [58.35269251730328]
  We introduce PyPulse, a Python package for imputation of biosignals in both clinical and wearable sensor settings.<n>PyPulse's framework provides a modular and extendable framework with high ease-of-use for a broad userbase, including non-machine-learning bioresearchers.<n>We released PyPulse under the MIT License on Github and PyPI.
  arXiv  Detail & Related papers  (2024-12-09T11:00:55Z)
• On Leakage of Code Generation Evaluation Datasets [44.4726918027046]
  We consider contamination by code generation test sets, in particular in their use in modern large language models. To address this, we release Less Basic Python Problems (LBPP): an uncontaminated new benchmark of 161 prompts with their associated Python solutions.
  arXiv  Detail & Related papers  (2024-07-10T11:50:20Z)
• SBFT Tool Competition 2024 -- Python Test Case Generation Track [4.149356993529412]
  Test case generation (TCG) for Python poses distinctive challenges due to the language's dynamic nature and the absence of strict type information. Previous research has successfully explored automated unit TCG for Python, with solutions outperforming random test generation methods. This paper describes our methodology, the analysis of the results together with the competing tools, and the challenges faced while running the competition experiments.
  arXiv  Detail & Related papers  (2024-01-26T20:21:15Z)
• A Static Evaluation of Code Completion by Large Language Models [65.18008807383816]
  Execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. static analysis tools such as linters, which can detect errors without running the program, haven't been well explored for evaluating code generation models. We propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees.
  arXiv  Detail & Related papers  (2023-06-05T19:23:34Z)
• Teaching Large Language Models to Self-Debug [62.424077000154945]
  Large language models (LLMs) have achieved impressive performance on code generation. We propose Self- Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations.
  arXiv  Detail & Related papers  (2023-04-11T10:43:43Z)
• Fault-Aware Neural Code Rankers [64.41888054066861]
  We propose fault-aware neural code rankers that can predict the correctness of a sampled program without executing it. Our fault-aware rankers can significantly increase the pass@1 accuracy of various code generation models.
  arXiv  Detail & Related papers  (2022-06-04T22:01:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.