Empirical Study on the Characteristics and Evolution of AI-usage in GitHub Repositories: Evidence from Code Comments
- URL: http://arxiv.org/abs/2606.06843v1
- Date: Fri, 05 Jun 2026 02:37:44 GMT
- Title: Empirical Study on the Characteristics and Evolution of AI-usage in GitHub Repositories: Evidence from Code Comments
- Abstract summary: We analyze 35,361 GitHub code comments that explicitly reference AI use and their associated code blocks.<n>We first open-code 500 unique comments and code blocks to derive a taxonomy of AI-assisted development activities.<n>We then annotate the full dataset using two LLM-based classifiers and aggregate predictions with Dawid-Skene expectation-maximization.
- Score: 7.213400161126317
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Developers increasingly use AI tools such as ChatGPT, Copilot, and Claude in everyday software workflows, but prior studies often evaluate LLM outputs in isolation rather than examining how developers adapt them in real projects. We analyze 35,361 GitHub code comments that explicitly reference AI use and their associated code blocks. We first open-code 500 unique comments and code blocks to derive a taxonomy of AI-assisted development activities, then annotate the full dataset using two LLM-based classifiers and aggregate predictions with Dawid-Skene expectation-maximization. We also analyze 12,996 subsequent commit messages to study how AI-assisted code evolves after introduction, and examine temporal trends from December 2022 to March 2026. Our results show that developers primarily use LLMs for code implementation, followed by code enhancement, debugging, documentation, and testing. Subsequent commits frequently involve refactoring and cleanup, feature integration and extension, and bug fixing, indicating sustained human oversight in adapting AI-assisted code. Over time, AI-referencing comments shift from direct code generation toward knowledge and conceptual support and code enhancement. These findings suggest that AI tools are becoming embedded not only as code-generation aids, but also as collaborative support mechanisms whose outputs are refined, extended, and corrected by developers over time.
Related papers
- How Developers Discuss Generative AI: A Longitudinal Study of the Visual Studio Code Community [0.0]
Generative AI tools such as GitHub Copilot, ChatGPT, and coding agents have rapidly become part of everyday software development.<n>This paper presents a longitudinal analysis of generative-AI-related discussions in the Visual Studio Code (VS Code) GitHub repository.
arXiv Detail & Related papers (2026-09-04T03:16:08Z) - How Do Software Professionals Evaluate AI-Generated Code? (Registered Report) [4.3772632092306045]
Recent advances in generative AI tools have significantly changed how software professionals write, evaluate, and interact with code.<n>Despite the growing adoption of and reliance on these tools, it remains unclear as to how software professionals evaluate the code they generate.<n>This research aims to build a theory of how software professionals evaluate AI-generated code, grounded in their accounts of evaluative practices, perceptions, and preferences.
arXiv Detail & Related papers (2026-07-10T14:01:00Z) - AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
arXiv Detail & Related papers (2026-05-18T17:08:26Z) - To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study [0.0]
This study empirically investigates the maintenance extent, human involvement, and modification types of AI-generated files versus human-authored code.<n>We analyzed over 1,000 files and approximately 3,200 changes from 100 popular repositories.
arXiv Detail & Related papers (2026-05-07T15:52:41Z) - RECAP: An End-to-End Platform for Capturing, Replaying, and Analyzing AI-Assisted Programming Interactions [21.75814643077238]
We present RECAP (Replay and Examine Captured AI Programming), an open-source platform that passively records AI chat sessions and fine-grained code edits inside VS Code without disrupting the developer's workflow.<n>We demonstrate how the platform's linked data and replay capabilities enable analyses of developer-AI interaction patterns that no single data source could support.
arXiv Detail & Related papers (2026-05-01T21:20:38Z) - SWE-chat: Coding Agent Interactions From Real Users in the Wild [70.18158706281724]
SWE-chat is the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild.<n>The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls.
arXiv Detail & Related papers (2026-04-22T17:08:19Z) - Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild [13.35459155979355]
We build a dataset of 304,362 verified AI-authored commits from 6,275 GitHub repositories.<n>We identify 484,606 distinct issues, and that code smells are by far the most common type, accounting for 89.1% of all issues.<n>These findings show that AI-generated code can introduce long-term maintenance costs into real software projects.
arXiv Detail & Related papers (2026-03-30T15:38:05Z) - Evolving with AI: A Longitudinal Analysis of Developer Logs [3.7353323067733473]
We study how sustained AI use reshapes actual daily coding practices in the long term.<n>We analyze five dimensions of workflow change: productivity, code quality, code editing, code reuse, and context switching.<n>Our results offer empirical insights into the silent restructuring of software and provide implications for designing future AI-augmented tooling.
arXiv Detail & Related papers (2026-01-15T10:30:24Z) - From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence [150.3696990310269]
Large language models (LLMs) have transformed automated software development by enabling direct translation of natural language descriptions into functional code.<n>We provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs.<n>We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder)
arXiv Detail & Related papers (2025-11-23T17:09:34Z) - Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [60.04362496037186]
We present the first controlled study of developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants.<n>Our results show agents can assist developers in ways that surpass copilots.
arXiv Detail & Related papers (2025-07-10T20:12:54Z) - Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework.
Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
arXiv Detail & Related papers (2024-10-02T09:11:10Z) - Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code.
We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
arXiv Detail & Related papers (2023-02-14T18:43:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.