The Hidden Costs of Automation: An Empirical Study on GitHub Actions Workflow Maintenance

• URL: http://arxiv.org/abs/2409.02366v1
• Date: Wed, 4 Sep 2024 01:33:16 GMT
• Title: The Hidden Costs of Automation: An Empirical Study on GitHub Actions Workflow Maintenance
• Authors: Pablo Valenzuela-Toledo, Alexandre Bergel, Timo Kehrer, Oscar Nierstrasz,
• Abstract summary: GitHub Actions (GA) is an orchestration platform that streamlines the automatic execution of engineering tasks. Human intervention is necessary to correct defects, update dependencies, or existing workflow files.
• Score: 45.53834452021771
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: GitHub Actions (GA) is an orchestration platform that streamlines the automatic execution of software engineering tasks such as building, testing, and deployment. Although GA workflows are the primary means for automation, according to our experience and observations, human intervention is necessary to correct defects, update dependencies, or refactor existing workflow files. In fact, previous research has shown that software artifacts similar to workflows, such as build files and bots, can introduce additional maintenance tasks in software projects. This suggests that workflow files, which are also used to automate repetitive tasks in professional software production, may generate extra workload for developers. However, the nature of such effort has not been well studied. This paper presents a large-scale empirical investigation towards characterizing the maintenance of GA workflows by studying the evolution of workflow files in almost 200 mature GitHub projects across ten programming languages. Our findings largely confirm the results of previous studies on the maintenance of similar artifacts, while also revealing GA-specific insights such as bug fixing and CI/CD improvement being among the major drivers of GA maintenance. A direct implication is that practitioners should be aware of proper resource planning and allocation for maintaining GA workflows, thus exposing the ``hidden costs of automation.'' Our findings also call for identifying and documenting best practices for such maintenance, and for enhanced tool features supporting dependency tracking and better error reporting of workflow specifications.

Related papers

• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorFBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures. We also present WorFEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms. We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? [73.81908518992161]
  We introduce Spider2-V, the first multimodal agent benchmark focusing on professional data science and engineering. Spider2-V features real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. These tasks evaluate the ability of a multimodal agent to perform data-related tasks by writing code and managing the GUI in enterprise data software systems.
  arXiv  Detail & Related papers  (2024-07-15T17:54:37Z)
• AutoCodeRover: Autonomous Program Improvement [8.66280420062806]
  We propose an automated approach for solving GitHub issues to autonomously achieve program improvement. In our approach called AutoCodeRover, LLMs are combined with sophisticated code search capabilities, ultimately leading to a program modification or patch. Experiments on SWE-bench-lite (300 real-life GitHub issues) show increased efficacy in solving GitHub issues (19% on SWE-bench-lite), which is higher than the efficacy of the recently reported SWE-agent.
  arXiv  Detail & Related papers  (2024-04-08T11:55:09Z)
• Automated User Story Generation with Test Case Specification Using Large Language Model [0.0]
  We developed a tool "GeneUS" to automatically create user stories from requirements documents. The output is provided in format leaving the possibilities open for downstream integration to the popular project management tools.
  arXiv  Detail & Related papers  (2024-04-02T01:45:57Z)
• On the effectiveness of Large Language Models for GitHub Workflows [9.82254417875841]
  Large Language Models (LLMs) have demonstrated their effectiveness in various software development tasks. We perform the first comprehensive study to understand the effectiveness of LLMs on five workflow-related tasks with different levels of prompts. Our evaluation of three state-of-art LLMs and their fine-tuned variants revealed various interesting findings on the current effectiveness and drawbacks of LLMs.
  arXiv  Detail & Related papers  (2024-03-19T05:14:12Z)
• Automated DevOps Pipeline Generation for Code Repositories using Large Language Models [5.011328607647701]
  The research scrutinizes the proficiency of GPT 3.5 and GPT 4 in generating GitHub, while assessing the influence of various prompt elements in constructing the most efficient pipeline. Results indicate substantial advancements in GPT 4. The research introduces a GitHub App built on Probot, empowering users to automate workflow generation within GitHub ecosystem.
  arXiv  Detail & Related papers  (2023-12-20T17:47:52Z)
• Reusability Challenges of Scientific Workflows: A Case Study for Galaxy [56.78572674167333]
  This study examined the reusability of existing and exposed several challenges. The challenges preventing reusability include tool upgrading, tool support, design flaws, incomplete, failure to load a workflow, etc.
  arXiv  Detail & Related papers  (2023-09-13T20:17:43Z)
• Toward Automatically Completing GitHub Workflows [16.302521048148748]
  We present GH-WCOM (GitHub COMpletion), a Transformer-based approach supporting developers in writing a specific type of CI/CD pipelines, namely GitHub. Our empirical study shows that GH-WCOM provides up to 34.23% correct predictions.
  arXiv  Detail & Related papers  (2023-08-31T14:53:00Z)
• Understanding the Challenges of Deploying Live-Traceability Solutions [45.235173351109374]
  SAFA.ai is a startup focusing on fine-tuning project-specific models that deliver automated traceability in a near real-time environment. This paper describes the challenges that characterize commercializing software traceability and highlights possible future directions.
  arXiv  Detail & Related papers  (2023-06-19T14:34:16Z)
• The GitHub Development Workflow Automation Ecosystems [47.818229204130596]
  Large-scale software development has become a highly collaborative endeavour. This chapter explores the ecosystems of development bots and GitHub Actions. It provides an extensive survey of the state-of-the-art in this domain.
  arXiv  Detail & Related papers  (2023-05-08T15:24:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.