The Impact of a Continuous Integration Service on the Delivery Time of Merged Pull Requests

• URL: http://arxiv.org/abs/2305.16365v1
• Date: Thu, 25 May 2023 10:59:35 GMT
• Title: The Impact of a Continuous Integration Service on the Delivery Time of Merged Pull Requests
• Abstract summary: We study whether adopting a CI service (TravisCI) can quicken the time to deliver merged PRs. Our results reveal that adopting a CI service may not necessarily quicken the delivery of merge PRs. The automation provided by CI and the boost in developers' confidence are key advantages of adopting a CI service.
• Score: 8.108605385023939
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continuous Integration (CI) is a software development practice that builds and tests software frequently (e.g., at every push). One main motivator to adopt CI is the potential to deliver software functionalities more quickly than not using CI. However, there is little empirical evidence to support that CI helps projects deliver software functionalities more quickly. Through the analysis of 162,653 pull requests (PRs) of 87 GitHub projects, we empirically study whether adopting a CI service (TravisCI) can quicken the time to deliver merged PRs. We complement our quantitative study by analyzing 450 survey responses from participants of 73 software projects. Our results reveal that adopting a CI service may not necessarily quicken the delivery of merge PRs. Instead, the pivotal benefit of a CI service is to improve the decision making on PR submissions, without compromising the quality or overloading the project's reviewers and maintainers. The automation provided by CI and the boost in developers' confidence are key advantages of adopting a CI service. Furthermore, open-source projects planning to attract and retain developers should consider the use of a CI service in their project, since CI is perceived to lower the contribution barrier while making contributors feel more confident and engaged in the project.

Related papers

• Studying Developer Perceptions on the Potential of CI Recommendation Systems [0.0]
  We aim to contact about 5,000 active GitHub developers, including both CI users and non-users.<n>The study investigates what drives CI adoption and service selection, distinguishing need-driven from socially influenced motivations.<n>Our findings will inform researchers developing CI recommendation systems and practitioners aiming to streamline CI adoption in open-source projects.
  arXiv  Detail & Related papers  (2026-08-03T01:42:17Z)
• Role of CI Adoption in Mobile App Success: An Empirical Study of Open-Source Android Projects [0.0]
  Continuous Integration (CI) helps automate builds, tests, and releases, but its impact on mobile development remains underexplored.<n> Existing studies mostly focus on CI adoption in general-purpose software.<n>We analyze open-source Android apps to compare CI adopters and non-adopters.
  arXiv  Detail & Related papers  (2026-02-02T11:03:38Z)
• The Promise and Reality of Continuous Integration Caching: An Empirical Study of Travis CI Builds [0.5951328229856059]
  Continuous Integration (CI) provides early feedback by automatically building software.<n>CI services offer caching mechanisms to speed up builds by reusing infrequently changing artifacts.<n>We analyze 513,384 builds from 1,279 GitHub projects.
  arXiv  Detail & Related papers  (2026-01-27T03:23:19Z)
• ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development [72.4729759618632]
  We introduce ABC-Bench, a benchmark to evaluate agentic backend coding within a realistic, executable workflow.<n>We curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories.<n>Our evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks.
  arXiv  Detail & Related papers  (2026-01-16T08:23:52Z)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents [79.29376673236142]
  Existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems.<n>We present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents.
  arXiv  Detail & Related papers  (2025-12-14T15:12:13Z)
• LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering [90.84806758077536]
  We introduce textbfLoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess large language models (LLMs) agents in realistic, long-context software engineering.<n>Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations.<n>Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens.
  arXiv  Detail & Related papers  (2025-11-17T23:57:24Z)
• From First Use to Final Commit: Studying the Evolution of Multi-CI Service Adoption [0.0]
  We analyze the historical CI adoption of 18,924 Java projects hosted on GitHub between January 2008 and December 2024.<n>Our analysis shows that the use of multiple CI services within the same project is a recurring pattern observed in nearly one in five projects.
  arXiv  Detail & Related papers  (2025-07-27T01:32:22Z)
• An Empirical Study of Complexity, Heterogeneity, and Compliance of GitHub Actions Workflows [0.0]
  GitHub Actions (GHA) has emerged as a dominant service due to its deep integration with GitHub.<n>GHA provides official documentation and community-supported best practices.<n>This study will investigate the structure, complexity, and compliance of GHA in open-source software repositories.
  arXiv  Detail & Related papers  (2025-07-24T03:26:38Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering [74.04271300772155]
  SyncMind is a framework that systematically defines the out-of-sync problem faced by large language model (LLM) agents in software engineering. Based on SyncMind, we create SyncBench, a benchmark featuring 24,332 instances of agent out-of-sync scenarios in real-world CSE.
  arXiv  Detail & Related papers  (2025-02-10T19:38:36Z)
• CI/CD Configuration Practices in Open-Source Android Apps: An Empirical Study [0.1433758865948252]
  We conduct an empirical study on Continuous Integration and Continuous Delivery practices in 2,564 Android apps. We observe a lack of commonality and standards across projects and services, leading to complex YML configurations. Our study emphasizes the necessity for automation and AI-powered tools to improve CI/CD processes for mobile applications.
  arXiv  Detail & Related papers  (2024-11-09T05:46:43Z)
• WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks [85.95607119635102]
  Large language models (LLMs) can mimic human-like intelligence. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents.
  arXiv  Detail & Related papers  (2024-07-07T07:15:49Z)
• Agent-Driven Automatic Software Improvement [55.2480439325792]
  This research proposal aims to explore innovative solutions by focusing on the deployment of agents powered by Large Language Models (LLMs) The iterative nature of agents, which allows for continuous learning and adaptation, can help surpass common challenges in code generation. We aim to use the iterative feedback in these systems to further fine-tune the LLMs underlying the agents, becoming better aligned to the task of automated software improvement.
  arXiv  Detail & Related papers  (2024-06-24T15:45:22Z)
• How to Understand Whole Software Repository? [64.19431011897515]
  An excellent understanding of the whole repository will be the critical path to Automatic Software Engineering (ASE) We develop a novel method named RepoUnderstander by guiding agents to comprehensively understand the whole repositories. To better utilize the repository-level knowledge, we guide the agents to summarize, analyze, and plan.
  arXiv  Detail & Related papers  (2024-06-03T15:20:06Z)
• Detecting Continuous Integration Skip : A Reinforcement Learning-based Approach [0.4297070083645049]
  Continuous Integration (CI) practices facilitate the seamless integration of code changes by employing automated building and testing processes. Some frameworks, such as Travis CI and GitHub Actions have significantly contributed to simplifying and enhancing the CI process. Developers continue to encounter difficulties in accurately flagging commits as either suitable for CI execution or as candidates for skipping.
  arXiv  Detail & Related papers  (2024-05-15T18:48:57Z)
• Practitioners' Challenges and Perceptions of CI Build Failure Predictions at Atlassian [9.781790288871]
  We report on an empirical study that investigates CI build failures throughout product development at Atlassian. Our quantitative analysis found that the repository dimension is the key factor influencing CI build failures. We found that the CI build prediction can not only provide proactive insight into CI build failures but also facilitate the team's decision-making.
  arXiv  Detail & Related papers  (2024-02-15T01:28:18Z)
• Experiential Co-Learning of Software-Developing Agents [83.34027623428096]
  Large language models (LLMs) have brought significant changes to various domains, especially in software development. We introduce Experiential Co-Learning, a novel LLM-agent learning framework. Experiments demonstrate that the framework enables agents to tackle unseen software-developing tasks more effectively.
  arXiv  Detail & Related papers  (2023-12-28T13:50:42Z)
• Code Ownership in Open-Source AI Software Security [18.779538756226298]
  We use code ownership metrics to investigate the correlation with latent vulnerabilities across five prominent open-source AI software projects. The findings suggest a positive relationship between high-level ownership (characterised by a limited number of minor contributors) and a decrease in vulnerabilities. With these novel code ownership metrics, we have implemented a Python-based command-line application to aid project curators and quality assurance professionals in evaluating and benchmarking their on-site projects.
  arXiv  Detail & Related papers  (2023-12-18T00:37:29Z)
• Embedded Software Development with Digital Twins: Specific Requirements for Small and Medium-Sized Enterprises [55.57032418885258]
  Digital twins have the potential for cost-effective software development and maintenance strategies. We interviewed SMEs about their current development processes. First results show that real-time requirements prevent, to date, a Software-in-the-Loop development approach.
  arXiv  Detail & Related papers  (2023-09-17T08:56:36Z)
• Empowered and Embedded: Ethics and Agile Processes [60.63670249088117]
  We argue that ethical considerations need to be embedded into the (agile) software development process. We put emphasis on the possibility to implement ethical deliberations in already existing and well established agile software development processes.
  arXiv  Detail & Related papers  (2021-07-15T11:14:03Z)
• A Brief Survey of Current Software Engineering Practices in Continuous Integration and Automated Accessibility Testing [0.0]
  This paper attempts to map the current scene of the software engineering effort spent on improving accessibility via continuous integration. We also try to explore steps that agile teams may take to train members on how to implement accessibility testing.
  arXiv  Detail & Related papers  (2021-02-27T01:13:43Z)
• Knowledge Integration of Collaborative Product Design Using Cloud Computing Infrastructure [65.2157099438235]
  The main focus of this paper is the concept of ongoing research in providing the knowledge integration service for collaborative product design and development using cloud computing infrastructure. Proposed knowledge integration services support users by giving real-time access to knowledge resources.
  arXiv  Detail & Related papers  (2020-01-16T18:44:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.