Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
- URL: http://arxiv.org/abs/2507.09089v2
- Date: Fri, 25 Jul 2025 00:43:07 GMT
- Title: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
- Authors: Joel Becker, Nate Rush, Elizabeth Barnes, David Rein,
- Abstract summary: Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied.<n>We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers.<n>16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience.
- Score: 0.5789840336223054
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. Each task is randomly assigned to allow or disallow usage of early 2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). To understand this result, we collect and evaluate evidence for 20 properties of our setting that a priori could contribute to the observed slowdown effect--for example, the size and quality standards of projects, or prior developer experience with AI tooling. Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.
Related papers
- Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [66.1850490474361]
We conduct the first academic study to explore developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants, GitHub Copilot and OpenHands.<n>Our results show agents have the potential to assist developers in ways that surpass copilots.
arXiv Detail & Related papers (2025-07-10T20:12:54Z) - SciMaster: Towards General-Purpose Scientific AI Agents, Part I. X-Master as Foundation: Can We Lead on Humanity's Last Exam? [51.112225746095746]
We introduce X-Master, a tool-augmented reasoning agent designed to emulate human researchers.<n>X-Masters sets a new state-of-the-art record on Humanity's Last Exam with a score of 32.1%.
arXiv Detail & Related papers (2025-07-07T17:50:52Z) - Echoes of AI: Investigating the Downstream Effects of AI Assistants on Software Maintainability [5.677464428950146]
This study investigates whether co-development with AI assistants affects software maintainability.<n> AI-assisted development in Phase 1 led to a modest speedup in subsequent evolution.<n>For habitual AI users, the mean speedup was 55.9%.
arXiv Detail & Related papers (2025-07-01T14:24:37Z) - How do Copilot Suggestions Impact Developers' Frustration and Productivity? [12.302518927205103]
We propose two theories on the impact of automatic suggestions on frustration and productivity.<n>We will involve at least 32 developers, both experts and novices.
arXiv Detail & Related papers (2025-04-09T11:55:22Z) - From Teacher to Colleague: How Coding Experience Shapes Developer Perceptions of AI Tools [0.0]
AI-assisted development tools promise productivity gains and improved code quality, yet their adoption among developers remains inconsistent.<n>We analyze survey data from 3380 developers to examine how coding experience relates to AI awareness, adoption, and the roles developers assign to AI in their workflow.
arXiv Detail & Related papers (2025-04-08T08:58:06Z) - How much does AI impact development speed? An enterprise-based randomized controlled trial [8.759453531975668]
We estimate the impact of three AI features on the time developers spent on a complex, enterprise-grade task.
We also found an interesting effect whereby developers who spend more hours on code-related activities per day were faster with AI.
arXiv Detail & Related papers (2024-10-16T18:31:14Z) - Does Co-Development with AI Assistants Lead to More Maintainable Code? A Registered Report [6.7428644467224]
This study aims to examine the influence of AI assistants on software maintainability.
In Phase 1, developers will add a new feature to a Java project, with or without the aid of an AI assistant.
In Phase 2, a randomized controlled trial, will involve a different set of developers evolving random Phase 1 projects - working without AI assistants.
arXiv Detail & Related papers (2024-08-20T11:48:42Z) - Consent in Crisis: The Rapid Decline of the AI Data Commons [74.68176012363253]
General-purpose artificial intelligence (AI) systems are built on massive swathes of public web data.
We conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora.
arXiv Detail & Related papers (2024-07-20T16:50:18Z) - Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code.
We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
arXiv Detail & Related papers (2023-02-14T18:43:34Z) - Seamful XAI: Operationalizing Seamful Design in Explainable AI [59.89011292395202]
Mistakes in AI systems are inevitable, arising from both technical limitations and sociotechnical gaps.
We propose that seamful design can foster AI explainability by revealing sociotechnical and infrastructural mismatches.
We explore this process with 43 AI practitioners and real end-users.
arXiv Detail & Related papers (2022-11-12T21:54:05Z) - Cybertrust: From Explainable to Actionable and Interpretable AI (AI2) [58.981120701284816]
Actionable and Interpretable AI (AI2) will incorporate explicit quantifications and visualizations of user confidence in AI recommendations.
It will allow examining and testing of AI system predictions to establish a basis for trust in the systems' decision making.
arXiv Detail & Related papers (2022-01-26T18:53:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.