Engineering Signals of Human-AI Collaboration in the Agentic Coding Era: A Longitudinal Analysis of 33,228 Pull Requests from vLLM and SGLang with Implications for Biomedical AI Agents and Bioinformatics Pipeline Developmen
- URL: http://arxiv.org/abs/2608.13884v1
- Date: Fri, 14 Aug 2026 02:22:28 GMT
- Title: Engineering Signals of Human-AI Collaboration in the Agentic Coding Era: A Longitudinal Analysis of 33,228 Pull Requests from vLLM and SGLang with Implications for Biomedical AI Agents and Bioinformatics Pipeline Developmen
- Abstract summary: We present a descriptive longitudinal analysis of seven engineering metrics: pull request (PR) throughput, cycle time, contributor diversity, PR comment density, merge rate, new-author participation, and PR size.<n>We segment development into four eras aligned with major changes in AI-assisted software development and examine human- and bot-authored activities.<n>Overall, AI-assisted development is associated with higher throughput, broader contributor participation, and increased AI-developer collaboration signals.
- Score: 0.7850626323769984
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The rapid adoption of AI coding assistants and autonomous agentic development systems has coincided with major changes in the pace and structure of open-source software engineering. Yet empirical longitudinal evidence of these changes at the team level remains limited. We present a descriptive longitudinal analysis of seven engineering metrics: pull request (PR) throughput, cycle time, contributor diversity, PR comment density, merge rate, new-author participation, and PR size. Metrics were computed from all merged PRs in two high-velocity AI infrastructure repositories, vLLM (February 2023-June 2026; 18,290 PRs) and SGLang (January 2024-June 2026; 14,938 PRs). We segment development into four eras aligned with major changes in AI-assisted software development and examine human- and bot-authored activities. Both projects show substantial increases in development velocity and AI-developer collaboration signals. PR throughput increased 21x in vLLM and 17.9x in SGLang, while bot-authored PRs accounted for less than 0.2% of this growth, indicating that the increase was overwhelmingly human-driven. In the latest era, median cycle time was 1.04 days for vLLM and 0.62 days for SGLang, while P90 cycle times reached 16.8 and 14.3 days, respectively. Monthly unique authors increased steadily in both projects, suggesting broader contributor participation. PR comment density increased 4.2x in vLLM and 3.8x in SGLang, with bot comments contributing an estimated 15-20% of the increase. In contrast, PR size remained relatively stable across eras. Overall, AI-assisted development is associated with higher throughput, broader contributor participation, and increased AI-developer collaboration signals in high-velocity open-source software development.
Related papers
- OpenAI4S: Code as Action, Science as Sessions [57.428071188574]
We present OpenAI4S, an open-source scientific research agent built around the principle of emphCode as Action, Science as Sessions.<n>OpenAI4S combines a persistent computing runtime with research-session management orchestration is handled through structured tool calls.<n>We evaluate OpenAI4S on 36 research scenarios spanning retrosynthesis, molecular protein binder design, protein mutation, catalyst screening, and mineral spectroscopy.
arXiv Detail & Related papers (2026-09-14T06:16:59Z) - AI-to-AI Code Reviews of GitHub Pull Requests [0.0]
Cross-product AI-to-AI review occurred in approximately 1.6% of identified agent-authored PRs.<n>For three of four dual-role reviewers, mean comments per PR were 58-65% higher in the same-product group.<n>Overall, closed-loop AI-to-AI review is increasing but remains a minority of identified agent activity.
arXiv Detail & Related papers (2026-08-21T17:17:35Z) - How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests [5.6795075579540395]
This study aims to characterize agentic pull requests (PR) in comparison to human generated PRs.<n>We first analyze how differences in merge rates between agentic and human generated PRs vary over time.<n>We then identify the types of development tasks where AI coding agents are predominantly applied.
arXiv Detail & Related papers (2026-07-23T21:48:28Z) - SWE-chat: Coding Agent Interactions From Real Users in the Wild [70.18158706281724]
SWE-chat is the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild.<n>The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls.
arXiv Detail & Related papers (2026-04-22T17:08:19Z) - AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub [0.0]
We present a large-scale dataset of merge conflicts in AI coding agent pull requests (Agentic PRs)<n>Our pipeline identifies 29K+ PRs exhibiting merge conflicts, yielding a conflict rate of 27.67%, and extracts 336K+ fine-grained conflict regions.<n>Our preliminary exploratory analysis indicates that merge conflicts are both frequent and often substantial in AI-generated contributions.
arXiv Detail & Related papers (2026-04-04T02:36:22Z) - ASI-Evolve: AI Accelerates AI [42.885465604426884]
ASI-Evolve is a framework for AI-for-AI research that closes the loop through a learn-design-experiment-analyze cycle.<n>To our knowledge, ASI-Evolve is the first unified framework to demonstrate AI-driven discovery across three central components of AI development.
arXiv Detail & Related papers (2026-03-31T12:04:17Z) - SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
arXiv Detail & Related papers (2026-03-25T19:26:44Z) - CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production [52.85500933801205]
CharacterFlywheel is an iterative process for improving large language models (LLMs) in production social chat applications.<n>We refined models across 15 generations using data from both internal and external real-user traffic.<n>We conducted 7-day A/B tests showing consistent engagement improvements.
arXiv Detail & Related papers (2026-03-02T15:27:31Z) - Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study [5.127121704630949]
We analyze 8,106 fix related PRs authored by five widely used AI coding agents from the AIDEV POP dataset.<n>Our results indicate that test case failures and prior resolution of the same issues by other PRs are the most common causes of non integration.
arXiv Detail & Related papers (2026-01-29T22:06:58Z) - On Autopilot? An Empirical Study of Human-AI Teaming and Review Practices in Open Source [11.412808537439973]
We investigated project-level guidelines and developers' interactions with AI-assisted pull requests (PRs)<n>We found that over 67.5% of AI-co-authored PRs originate from contributors without prior code ownership.<n>In contrast to human-created PRs where non-owner developers receive the most feedback, AI-co-authored PRs from non-owners receive the least.
arXiv Detail & Related papers (2026-01-20T09:09:53Z) - The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering [10.252332355171237]
This paper introduces AIDev, the first largescale dataset capturing how such agents operate in the wild.<n>Spanning over 456,000 pull requests by five leading agents, AIDev provides an unprecedented empirical foundation for studying autonomous teammates in software development.<n>The dataset includes rich on PRs, authorship, review timelines, code changes, and integration outcomes.
arXiv Detail & Related papers (2025-07-20T15:15:58Z) - OpenHands: An Open Platform for AI Software Developers as Generalist Agents [109.8507367518992]
We introduce OpenHands, a platform for the development of AI agents that interact with the world in similar ways to a human developer.<n>We describe how the platform allows for the implementation of new agents, safe interaction with sandboxed environments for code execution, and incorporation of evaluation benchmarks.
arXiv Detail & Related papers (2024-07-23T17:50:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.