Where did we fail? -- Reproducing build failures in embedded open source software

• URL: http://arxiv.org/abs/2604.27075v1
• Date: Wed, 29 Apr 2026 18:07:04 GMT
• Title: Where did we fail? -- Reproducing build failures in embedded open source software
• Abstract summary: We present PhantomRun, a dataset that standardizes the retrieval, storage, and reproduction of continuous integration build logs and metadata.<n>Across 4628 failing CI runs, we reconstructed 91.8% of builds and preserved execution outcomes in 98% of evaluated cases.<n>An empirical evaluation shows that reproduced builds closely match their originals, typically differing only in timestamps or minor nondeterministic reordering.
• Score: 2.64399132991614
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Due to hardware-software co-development in embedded systems, continuous integration (CI) builds frequently fail because of complex cross-compilation, board configurations, and toolchain constraints. Although CI build logs contain valuable diagnostic information, they are short-lived and difficult to reuse due to heterogeneous runners, toolchains, and log formats. To address these challenges, we present PhantomRun, a unified abstraction layer and publicly reusable dataset that standardizes the retrieval, storage, and reproduction of CI build logs and metadata. Across 4628 failing CI runs, we reconstructed 91.8% of builds and preserved execution outcomes in 98% of evaluated cases. PhantomRun provides two core capabilities: retrieving the build log of any commit and faithfully re-executing the corresponding build in a controlled environment. By exposing all build artifacts and metadata in a uniform, machine-readable format, PhantomRun enables reproducible and longitudinal studies of CI failures. An empirical evaluation shows that reproduced builds closely match their originals, typically differing only in timestamps or minor nondeterministic reordering, demonstrating the feasibility of large-scale historical CI reconstruction.

Related papers

• LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent [60.27239351179265]
  LOOPSBENCH is a benchmark for loop engineering in coding agent evaluation.<n>It comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains.<n>Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations.
  arXiv  Detail & Related papers  (2026-07-31T20:18:25Z)
• Taming the Drift: Context-aware Repair of Dockerfile Drift during Software Evolution [52.50730821321986]
  Cadre is a context-aware framework for automated Dockerfile drift repair.<n>CDG maps each Dockerfile instruction to its file-level and inter-instruction dependencies.<n>DodeX mines real-world Dockerfile drift instances from GitHub Actions CI logs.
  arXiv  Detail & Related papers  (2026-07-14T09:12:26Z)
• SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers [55.39407031861402]
  We introduce SCDBench, a dataset and benchmark methodology for smart contract decompilation.<n>The dataset contains 600 real-world Solidity contracts with paired bytecode inputs, ground-truth source code, and replayable semantic checkpoints.<n>We evaluate Claude Opus 4.7, GPT-5.3-Codex, and GLM-5 in a zero-shot decompilation setting, including GLM-5 variants with and without extended reasoning.
  arXiv  Detail & Related papers  (2026-05-27T20:08:47Z)
• Understanding by Reconstruction: Reversing the Software Development Process for LLM Pretraining [66.89012795621349]
  Large Language Models (LLMs) often struggle with the deep, long-horizon reasoning required for complex software engineering.<n>We propose a novel paradigm: understanding via reconstruction.<n>We introduce a framework that synthesizes latent agentic trajectories using a multi-agent simulation.
  arXiv  Detail & Related papers  (2026-03-11T09:23:20Z)
• His2Trans: A Skeleton First Framework for Self Evolving C to Rust Translation with Historical Retrieval [45.246293154277886]
  His2Trans is a framework that combines a deterministic, build-aware skeleton with self-evolving knowledge extraction to support stable, incremental migration.<n> Experiments on industrial OpenHarmony modules show that His2Trans reaches a 99.75% incremental compilation pass rate.
  arXiv  Detail & Related papers  (2026-03-03T05:42:08Z)
• PhantomRun: Auto Repair of Compilation Errors in Embedded Open Source Software [2.64399132991614]
  We study four major open-source embedded system projects, spanning over 4000 build failures from the project's CI runs.<n>We find that hardware dependencies account for the majority of compilation failures, followed by syntax errors and build-script issues.<n>We present PhantomRun, an automated framework that leverages large language models (LLMs) to generate and validate fixes for CI compilation failures.
  arXiv  Detail & Related papers  (2026-02-23T19:13:22Z)
• Understanding and Detecting Flaky Builds in GitHub Actions [6.3850400710838615]
  We present a large-scale empirical study of flaky builds in GitHub Actions based on rerun data from 1,960 Java projects.<n>We identify 15 distinct categories of flaky failures, among which flaky tests, network issues, and dependency resolution issues are the most prevalent.<n>We propose a machine learning-based approach for detecting flaky failures at the job level.
  arXiv  Detail & Related papers  (2026-02-02T16:39:56Z)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents [79.29376673236142]
  Existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems.<n>We present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents.
  arXiv  Detail & Related papers  (2025-12-14T15:12:13Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• Attestable Builds: Compiling Verifiable Binaries on Untrusted Systems using Trusted Execution Environments [2.4650753804485417]
  We present attestable builds, a new paradigm to provide strong source-to-binary correspondence in software artifacts.<n>Our system uses modern trusted execution environments (TEEs) and sandboxed build containers to provide strong guarantees that a given artifact was correctly built from a specific source code snapshot.
  arXiv  Detail & Related papers  (2025-05-05T10:00:04Z)
• Does Functional Package Management Enable Reproducible Builds at Scale? Yes [4.492444446637857]
  Reproducible Builds (R-B) guarantee that rebuilding a software package from source leads to bitwise identical artifacts.<n>We perform the first large-scale study of bitwise in the context of the Nix functional package manager.<n>We obtain very high bitwise rates, between 69 and 91% with an upward trend, and even higher rebuildability rates, over 99%.
  arXiv  Detail & Related papers  (2025-01-27T10:11:27Z)
• On the Security Blind Spots of Software Composition Analysis [46.1389163921338]
  We present a novel approach to detect vulnerable clones in the Maven repository. We retrieve over 53k potential vulnerable clones from Maven Central. We detect 727 confirmed vulnerable clones and synthesize a testable proof-of-vulnerability project for each of those.
  arXiv  Detail & Related papers  (2023-06-08T20:14:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.