Is this Build Failure Related to my Patch? An Empirical Study of Unrelated Build Failures in Continuous Integration

• URL: http://arxiv.org/abs/2605.05564v1
• Date: Thu, 07 May 2026 01:13:14 GMT
• Title: Is this Build Failure Related to my Patch? An Empirical Study of Unrelated Build Failures in Continuous Integration
• Abstract summary: We study 77,354 Continuous Integration (CI) build failures from seven Apache projects to understand and predict unrelated build failures.<n>We find that developers spend a median of 4 hours identifying whether a failure is related or unrelated to their push.
• Score: 0.32665457005470505
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continuous Integration (CI) systems often run many builds concurrently. In this setting, a legitimate build failure may not be caused by the code push that triggered it. Such unrelated build failures can waste developer effort because developers must determine whether the failure is actionable for their current change. We study 77,354 CI build failures from seven open source Apache projects to understand and predict unrelated build failures. We find that developers spend a median of 4 hours identifying whether a failure is related or unrelated to their push. We also perform a document analysis of 371 confirmed unrelated build failures sampled from 10,316 potentially unrelated failures. The analysis shows that unrelated test failures account for 20% of the cases in which developers classify build failures as unrelated. To predict unrelated build failures, we extract 33 features from issue reports, issue comments, and commits associated with the triggering push. We build semi-supervised Positive and Unlabeled (PU) learning models for seven Apache projects. The models achieve precision from 0.70 to 0.88, recall from 0.30 to 1.00, F1-score from 0.44 to 0.91, and AUC from 0.63 to 0.97. Feature importance analysis shows that CI latency, repeated error messages, and the number of preceding comments are useful indicators of unrelated build failures. These results show that PU learning can help developers identify build failures that are unlikely to be caused by their current push.

Related papers

• FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs [46.58695049126903]
  We present FuzzingBrain-Bench, a benchmark for assessing AI models' ability to discover bugs in open-source software.<n>A model's performance on each challenge is scored based on the number of distinct crash signatures it produces.<n>Claude Opus 4.8 performs best, triggering crashes in 60 of 77 challenges and achieving a score of 196 out of 579.
  arXiv  Detail & Related papers  (2026-08-25T21:09:20Z)
• Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes [55.114939648705395]
  Large language models (LLMs) have made automated program repair increasingly practical for real-world bugs.<n>Bug reproduction tests (BRTs) help close this gap by turning a bug report into an executable, bug-specific signal.<n>We propose CoHarden, a co-generation framework that uses the Lax signal as an in-loop convergence criterion.
  arXiv  Detail & Related papers  (2026-07-22T07:30:07Z)
• AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents [83.37969790806069]
  LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.<n>We present Agent DebuggerX, an open-source framework that organizes debug as a closed loop of Detect, Attribute, Recover, and Rerun.<n>At its core, Deep Debugger performs multi-turn root-cause diagnosis through global trajectory understanding.
  arXiv  Detail & Related papers  (2026-07-21T06:21:13Z)
• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• Where did we fail? -- Reproducing build failures in embedded open source software [2.64399132991614]
  We present PhantomRun, a dataset that standardizes the retrieval, storage, and reproduction of continuous integration build logs and metadata.<n>Across 4628 failing CI runs, we reconstructed 91.8% of builds and preserved execution outcomes in 98% of evaluated cases.<n>An empirical evaluation shows that reproduced builds closely match their originals, typically differing only in timestamps or minor nondeterministic reordering.
  arXiv  Detail & Related papers  (2026-04-29T18:07:04Z)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
  We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
  arXiv  Detail & Related papers  (2026-03-25T19:26:44Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• PhantomRun: Auto Repair of Compilation Errors in Embedded Open Source Software [2.64399132991614]
  We study four major open-source embedded system projects, spanning over 4000 build failures from the project's CI runs.<n>We find that hardware dependencies account for the majority of compilation failures, followed by syntax errors and build-script issues.<n>We present PhantomRun, an automated framework that leverages large language models (LLMs) to generate and validate fixes for CI compilation failures.
  arXiv  Detail & Related papers  (2026-02-23T19:13:22Z)
• Understanding and Detecting Flaky Builds in GitHub Actions [6.3850400710838615]
  We present a large-scale empirical study of flaky builds in GitHub Actions based on rerun data from 1,960 Java projects.<n>We identify 15 distinct categories of flaky failures, among which flaky tests, network issues, and dependency resolution issues are the most prevalent.<n>We propose a machine learning-based approach for detecting flaky failures at the job level.
  arXiv  Detail & Related papers  (2026-02-02T16:39:56Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• On the Illusion of Success: An Empirical Study of Build Reruns and Silent Failures in Industrial CI [1.2744523252873348]
  This paper presents the first empirical study of silent failures through the practice of rerunning successful jobs.<n>An analysis of 142,387 jobs across 81 industrial projects shows that 11% of successful jobs are rerun, with 35% of these reruns occurring after more than 24 hours.<n>Key factors associated with reruns of successful jobs include testing and static analysis tasks, scripting languages like Shell, and developers prior rerun tendencies.
  arXiv  Detail & Related papers  (2025-09-17T18:26:29Z)
• Do Large Language Model Benchmarks Test Reliability? [66.1783478365998]
  We investigate how well current benchmarks quantify model reliability.<n>Motivated by this gap in the evaluation of reliability, we propose the concept of so-called platinum benchmarks.<n>We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks.
  arXiv  Detail & Related papers  (2025-02-05T18:58:19Z)
• 230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers [9.45325012281881]
  Flaky tests are tests that can non-deterministically pass or fail, even in the absence of code changes. How to quickly determine if a test failed due to flakiness, or if it detected a bug?
  arXiv  Detail & Related papers  (2024-01-28T22:36:30Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.