Who Writes the Docs in SE 3.0? Agent vs. Human Documentation Pull Requests

• URL: http://arxiv.org/abs/2601.20171v1
• Date: Wed, 28 Jan 2026 02:11:34 GMT
• Title: Who Writes the Docs in SE 3.0? Agent vs. Human Documentation Pull Requests
• Abstract summary: We analyze 1,997 documentation-related pull requests (PRs) authored by AI agents and human developers.<n>We find that AI agents submit substantially more documentation-related PRs than humans in the studied repositories.
• Score: 3.23966298972006
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As software engineering moves toward SE3.0, AI agents are increasingly used to carry out development tasks and contribute changes to software projects. It is therefore important to understand the extent of these contributions and how human developers review and intervene, since these factors shape the risks of delegating work to AI agents. While recent studies have examined how AI agents support software development tasks (e.g., code generation, issue resolution, and PR automation), their role in documentation tasks remains underexplored-even though documentation is widely consumed and shapes how developers understand and use software. Using the AIDev, we analyze 1,997 documentation-related pull requests (PRs) authored by AI agents and human developers, where documentation PRs are those that create or modify project documentation artifacts. We find that AI agents submit substantially more documentation-related PRs than humans in the studied repositories. We further observe that agent-authored documentation edits are typically integrated with little follow-up modification from humans, raising concerns about review practices and the reliability of agent-generated documentation. Overall, while AI agents already contribute substantially to documentation workflows, our results suggest concerns for emerging challenges for documentation quality assurance and human-AI collaboration in SE3.0.

Related papers

• Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs [0.991629944808926]
  We conduct a comparative analysis of 7,191 agent-generated PRs with 1402 human-authored PRs from Python repositories.<n>We show that AI agents introduce fewer breaking changes overall than humans (3.45% vs. 7.40%) in code generation tasks.<n>Agents exhibit substantially higher risk during maintenance tasks, with changes introducing breaking changes at rates of 6.72% and 9.35%, respectively.
  arXiv  Detail & Related papers  (2026-03-29T05:15:13Z)
• Supporting software engineering tasks with agentic AI: Demonstration on document retrieval and test scenario generation [0.0]
  We introduce agentic AI solutions for two software engineering tasks.<n>First, we developed a solution for automatic test scenario generation from a detailed requirements description.<n>Second, we developed an agentic AI solution for the document retrieval task in the context of software engineering documents.
  arXiv  Detail & Related papers  (2026-02-04T16:33:16Z)
• Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agents [4.744786007044749]
  We study how core and peripheral developers use, review, modify, and verify agent-generated contributions prior to acceptance.<n>A subset of peripheral developers use agents more often, delegating tasks evenly across bug fixing, feature addition, documentation, and testing.<n>In contrast, core developers focus more on documentation and testing, yet their agentic PRs are frequently merged into the main/master branch.
  arXiv  Detail & Related papers  (2026-01-27T22:50:01Z)
• On Autopilot? An Empirical Study of Human-AI Teaming and Review Practices in Open Source [11.412808537439973]
  We investigated project-level guidelines and developers' interactions with AI-assisted pull requests (PRs)<n>We found that over 67.5% of AI-co-authored PRs originate from contributors without prior code ownership.<n>In contrast to human-created PRs where non-owner developers receive the most feedback, AI-co-authored PRs from non-owners receive the least.
  arXiv  Detail & Related papers  (2026-01-20T09:09:53Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub [6.7302091035327285]
  Large language models (LLMs) are increasingly being integrated into software development processes.<n>The ability to generate code and submit pull requests with minimal human intervention, through the use of autonomous AI agents, is poised to become a standard practice.<n>We empirically study 567 GitHub pull requests (PRs) generated using Claude Code, an agentic coding tool, across 157 open-source projects.
  arXiv  Detail & Related papers  (2025-09-18T08:48:32Z)
• The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering [10.252332355171237]
  This paper introduces AIDev, the first largescale dataset capturing how such agents operate in the wild.<n>Spanning over 456,000 pull requests by five leading agents, AIDev provides an unprecedented empirical foundation for studying autonomous teammates in software development.<n>The dataset includes rich on PRs, authorship, review timelines, code changes, and integration outcomes.
  arXiv  Detail & Related papers  (2025-07-20T15:15:58Z)
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [60.04362496037186]
  We present the first controlled study of developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants.<n>Our results show agents can assist developers in ways that surpass copilots.
  arXiv  Detail & Related papers  (2025-07-10T20:12:54Z)
• SciMaster: Towards General-Purpose Scientific AI Agents, Part I. X-Master as Foundation: Can We Lead on Humanity's Last Exam? [51.112225746095746]
  We introduce X-Master, a tool-augmented reasoning agent designed to emulate human researchers.<n>X-Masters sets a new state-of-the-art record on Humanity's Last Exam with a score of 32.1%.
  arXiv  Detail & Related papers  (2025-07-07T17:50:52Z)
• FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks [52.47895046206854]
  FieldWorkArena is a benchmark for agentic AI targeting real-world field work.<n>This paper defines a new action space that agentic AI should possess for real world work environment benchmarks.
  arXiv  Detail & Related papers  (2025-05-26T08:21:46Z)
• DocAgent: A Multi-Agent System for Automated Code Documentation Generation [7.653779364214401]
  We introduce DocAgent, a novel multi-agent collaborative system using topological code processing for incremental context building.<n>Specialized agents (Reader, Searcher, Writer, Verifier, Orchestrator) then collaboratively generate documentation.<n>We also propose a multi-faceted evaluation framework assessing Completeness, Helpfulness, and Truthfulness.
  arXiv  Detail & Related papers  (2025-04-11T17:50:08Z)
• OpenHands: An Open Platform for AI Software Developers as Generalist Agents [109.8507367518992]
  We introduce OpenHands, a platform for the development of AI agents that interact with the world in similar ways to a human developer.<n>We describe how the platform allows for the implementation of new agents, safe interaction with sandboxed environments for code execution, and incorporation of evaluation benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T17:50:43Z)
• A Methodology for Creating AI FactSheets [67.65802440158753]
  This paper describes a methodology for creating the form of AI documentation we call FactSheets. Within each step of the methodology, we describe the issues to consider and the questions to explore. This methodology will accelerate the broader adoption of transparent AI documentation.
  arXiv  Detail & Related papers  (2020-06-24T15:08:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.