On the Illusion of Success: An Empirical Study of Build Reruns and Silent Failures in Industrial CI

• URL: http://arxiv.org/abs/2509.14347v1
• Date: Wed, 17 Sep 2025 18:26:29 GMT
• Title: On the Illusion of Success: An Empirical Study of Build Reruns and Silent Failures in Industrial CI
• Abstract summary: This paper presents the first empirical study of silent failures through the practice of rerunning successful jobs.<n>An analysis of 142,387 jobs across 81 industrial projects shows that 11% of successful jobs are rerun, with 35% of these reruns occurring after more than 24 hours.<n>Key factors associated with reruns of successful jobs include testing and static analysis tasks, scripting languages like Shell, and developers prior rerun tendencies.
• Score: 1.2744523252873348
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reliability of build outcomes is a cornerstone of effective Continuous Integration (CI). Yet in practice, developers often struggle with non-deterministic issues in the code or CI infrastructure, which undermine trust in build results. When faced with such unexpected outcomes, developers often repeatedly rerun jobs hoping for true success, but this practice is known to increase CI costs and reduce productivity. While recent studies have focused on intermittent job failures, no prior work has investigated silent failures, where build jobs are marked as successful but fail to complete all or part of their tasks. Such silent failures often go unnoticed, creating an illusion of success with detrimental consequences such as bugs escaping into production. This paper presents the first empirical study of silent failures through the practice of rerunning successful jobs. An analysis of 142,387 jobs across 81 industrial projects shows that 11% of successful jobs are rerun, with 35% of these reruns occurring after more than 24 hours. Using mixed-effects models on 32 independent variables (AUC of 85%), we identified key factors associated with reruns of successful jobs, notably testing and static analysis tasks, scripting languages like Shell, and developers prior rerun tendencies. A further analysis of 92 public issues revealed 11 categories of silent failures aligning with these factors, the most frequent being artifact operation errors, caching errors, and ignored exit codes. Overall, our findings provide valuable insights into the circumstances and causes of silent failures to raise awareness among teams, and present solutions to improve CI reliability.

Related papers

• Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
  Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
  arXiv  Detail & Related papers  (2026-08-14T07:26:38Z)
• Failure as a Process: An Anatomy of CLI Coding Agent Trajectories [21.513740407266354]
  Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments.<n>Existing empirical studies investigate why coding agents fail, yet they largely treat failure as a final outcome rather than a temporal process.<n>We present the first large-scale empirical study of CLI coding-agent failure trajectories, introducing a process-oriented framework that analyzes failure through its onset, evolution, and recovery across execution trajectories.
  arXiv  Detail & Related papers  (2026-07-10T15:25:56Z)
• Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents [80.03824805228719]
  A major challenge in developing computer-use agents is collecting large-scale, high-quality trajectories.<n>We propose a data-centric paradigm that turns failed trajectories into agent improvements.<n>We validate this approach with the state-of-the-art OpenCUA-72B model on the OSWorld benchmark.
  arXiv  Detail & Related papers  (2026-06-30T07:44:37Z)
• When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime [4.56904471801595]
  We present a longitudinal study of silent failures in a personal-assistant agent runtime in continuous production since March 2026.<n>We documented 22 incidents with full root-cause postmortems, in which one meta-pattern manifested at least 28 times.<n>We derive a five-class, mechanism-oriented taxonomy: (A) environment and platform quirks, (B) design-assumption mismatches, (C) error swallowing and dilution, (D) chained hallucination and fabrication, (E) operational omission and forensic blind spots.
  arXiv  Detail & Related papers  (2026-06-12T16:06:55Z)
• DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policies [65.27025563507961]
  We propose DreamAvoid, a critical-phase test-time dreaming framework for Vision-Language-Action (VLA) models.<n>We also introduce an autonomous boundary learning paradigm to refine the system's understanding of the subtle boundary between success and failure.<n>The results demonstrate that DreamAvoid can effectively avoid failures, thereby improving the overall task success rate.
  arXiv  Detail & Related papers  (2026-05-12T08:27:16Z)
• A Rubric-Supervised Critic from Sparse Real-World Outcomes [87.11204512676193]
  Real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse.<n>We propose a process to learn a "critic" model from sparse and noisy interaction data, which can then be used both as a reward model for either RL-based training or inference-time scaling.
  arXiv  Detail & Related papers  (2026-03-04T07:23:54Z)
• When Is Enough Not Enough? Illusory Completion in Search Agents [56.98225130959051]
  We study whether search agents reliably reason across all requirements by tracking, verifying, and maintaining multiple conditions.<n>We find that illusory completion frequently occurs, wherein agents believe tasks are complete despite unresolved or violated constraints, leading to underverified answers.<n>We examine whether explicit constraint-state tracking during execution mitigates these failures via LiveLedger, an inference-time tracker.
  arXiv  Detail & Related papers  (2026-02-07T13:50:38Z)
• Failure-Aware Enhancements for Large Language Model (LLM) Code Generation: An Empirical Study on Decision Framework [0.26508608365976566]
  In an empirical study of 25 GitHub projects, we found that progressive prompting achieves 96.9% average task completion.<n>Self-critique succeeds on code-reviewable logic errors but fails completely on external service integration.<n>RAG achieves highest completion across all failure types with superior efficiency.
  arXiv  Detail & Related papers  (2026-02-02T23:08:03Z)
• Understanding and Detecting Flaky Builds in GitHub Actions [6.3850400710838615]
  We present a large-scale empirical study of flaky builds in GitHub Actions based on rerun data from 1,960 Java projects.<n>We identify 15 distinct categories of flaky failures, among which flaky tests, network issues, and dependency resolution issues are the most prevalent.<n>We propose a machine learning-based approach for detecting flaky failures at the job level.
  arXiv  Detail & Related papers  (2026-02-02T16:39:56Z)
• Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models [1.2744523252873348]
  FlaXifyer is a few-shot learning approach for predicting intermittent job failure categories using pre-trained language models.<n>LogSift is an interpretability technique that identifies influential log statements in under one second.<n> Evaluation on 2,458 job failures from TELUS demonstrates that FlaXifyer and LogSift enable effective automated triage, accelerate failure diagnosis, and pave the way towards the automated resolution of intermittent job failures.
  arXiv  Detail & Related papers  (2026-01-29T19:34:34Z)
• Measuring Agents in Production [133.77818981073457]
  We present the first large-scale systematic study of AI agents in production.<n>We find that production agents are typically built using simple, controllable approaches.<n> Reliability remains the top development challenge, driven by difficulties in ensuring and evaluating agent correctness.
  arXiv  Detail & Related papers  (2025-12-02T16:45:10Z)
• SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents [52.20768003832476]
  We analyze execution traces on $$-Bench (Airline/Retail) and SWE-Bench Verified.<n>We formalize emphdecisive deviations, earliest action, level divergences that flip success to failure.<n>We introduce cm, a model-agnostic, gradient-free, test-time safeguard.
  arXiv  Detail & Related papers  (2025-11-26T01:28:22Z)
• Efficient Detection of Intermittent Job Failures Using Few-Shot Learning [2.8402080392117757]
  We introduce a novel approach to intermittent job failure detection using few-shot learning.<n>Our approach achieves 70-88% F1-score with only 12 shots in all projects, outperforming the state-of-the-art (SOTA) approach.
  arXiv  Detail & Related papers  (2025-07-05T22:04:01Z)
• Is there a half-life for the success rates of AI agents? [0.0]
  I show that the performance of AI agents on longer-duration tasks can be explained by a constant rate of failing during each minute a human would take to do the task.<n>This implies an exponentially declining success rate with the length of the task and that each agent could be characterised by its own half-life.
  arXiv  Detail & Related papers  (2025-05-08T10:31:03Z)
• Exploring Expert Failures Improves LLM Agent Tuning [74.0772570556016]
  We propose Exploring Expert Failures (EEF), which identifies beneficial actions from failed expert trajectories.<n>EEF successfully solves some previously unsolvable subtasks and improves agent tuning performance.
  arXiv  Detail & Related papers  (2025-04-17T17:53:54Z)
• SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories [55.161075901665946]
  Super aims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges, and 602 automatically generated problems for larger-scale development. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios.
  arXiv  Detail & Related papers  (2024-09-11T17:37:48Z)
• Practitioners' Challenges and Perceptions of CI Build Failure Predictions at Atlassian [9.781790288871]
  We report on an empirical study that investigates CI build failures throughout product development at Atlassian. Our quantitative analysis found that the repository dimension is the key factor influencing CI build failures. We found that the CI build prediction can not only provide proactive insight into CI build failures but also facilitate the team's decision-making.
  arXiv  Detail & Related papers  (2024-02-15T01:28:18Z)
• Competition-Level Problems are Effective LLM Evaluators [121.15880285283116]
  This paper aims to evaluate the reasoning capacities of large language models (LLMs) in solving recent programming problems in Codeforces. We first provide a comprehensive evaluation of GPT-4's peiceived zero-shot performance on this task, considering various aspects such as problems' release time, difficulties, and types of errors encountered. Surprisingly, theThoughtived performance of GPT-4 has experienced a cliff like decline in problems after September 2021 consistently across all the difficulties and types of problems.
  arXiv  Detail & Related papers  (2023-12-04T18:58:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.