An Exploratory Study on LLM-Generated Code and Comments in Code Repositories

• URL: http://arxiv.org/abs/2607.01867v1
• Date: Thu, 02 Jul 2026 08:25:15 GMT
• Title: An Exploratory Study on LLM-Generated Code and Comments in Code Repositories
• Abstract summary: We study the code and comments detected as likely to be generated by LLMs and their characteristics.<n>We conduct experiments on active company- and community-maintained repositories from 2021 to 2025.
• Score: 6.161007996548193
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The use of LLMs in software development has become increasingly widespread on tasks such as code generation and summarization. Reports from large technology companies showed that around 20% to 30% of their code are generated by LLMs. However, there remains skepticism about the practical usage of LLM-generated code and comments, such as concerns on more time for debugging the generated code and the unnaturalness of the generated comments. In this paper, we study the code and comments detected as likely to be generated by LLMs and their characteristics, the differences between company- and community-maintained repositories, and how likely bugs are associated with LLM-generated code. We conduct extensive experiments on active company- and community-maintained repositories from 2021 to 2025 using various tools and techniques that detect code and comments generated by LLMs. Based on our detector-based proxy analysis, the results suggest that code detected as likely to be generated by LLMs decreased over time and appeared frequently in test cases, while that of comments remains relatively stable. Proxy results further suggest that code detected as likely to be generated by LLMs shows substantial intra-repository code clones, whereas comments exhibit a relatively low proportion of grammatically correct sentences. In addition, the company-maintained repositories show a higher percentage of code and comments detected as likely to be generated by LLMs, and only a small percentage of the human-labelled bugs are detected as being likely associated with LLM-generated code.

Related papers

• Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code [3.7910079742785787]
  We conduct a Wizard-of-Oz experiment to examine how software engineers behave when code is explicitly labeled as LLM-generated during a code review task.<n>We found that while the thoroughness of code review did not change for participants, they spent more time fixating on LLM-labelled code, indicating that the label itself influences attention.<n> Practitioners also adapted their review strategy for LLM-labelled code by assessing the code based on specific criteria, or using the prompt to guide their review.
  arXiv  Detail & Related papers  (2026-06-25T01:23:05Z)
• RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices [54.956760584923295]
  Researchers have made substantial progress using Large Language Models (LLMs) for code generation.<n>However, developers typically write code based on structured designs or specifications rather than raw natural language descriptions.<n>This gap between existing benchmarks and real industry development practices means that current benchmark scores may not accurately reflect how much code generation can help automate development tasks.
  arXiv  Detail & Related papers  (2026-04-24T15:35:54Z)
• The Fools are Certain; the Wise are Doubtful: Exploring LLM Confidence in Code Completion [4.215010577170175]
  We evaluate the confidence of Large Language Models (LLMs) when generating code by measuring code perplexity.<n>We find that strongly-typed languages exhibit lower perplexity than dynamically typed languages.<n> Perl appears universally high in perplexity, whereas Java appears low.
  arXiv  Detail & Related papers  (2025-08-22T06:51:13Z)
• IFEvalCode: Controlled Code Generation [69.28317223249358]
  The paper introduces forward and backward constraints generation to improve the instruction-following capabilities of Code LLMs.<n>The authors present IFEvalCode, a multilingual benchmark comprising 1.6K test samples across seven programming languages.
  arXiv  Detail & Related papers  (2025-07-30T08:08:48Z)
• Code Evolution Graphs: Understanding Large Language Model Driven Design of Algorithms [2.827573861233375]
  Large Language Models (LLMs) have demonstrated great promise in generating code.<n>We show results for three benchmark problem classes and demonstrate novel insights.
  arXiv  Detail & Related papers  (2025-03-20T19:30:22Z)
• Combining LLM Code Generation with Formal Specifications and Reactive Program Synthesis [0.7580487359358722]
  Large Language Models (LLMs) struggle with accuracy and are unsuitable for high-risk applications. We introduce a solution that divides the code generation into two parts; one to be handled by an LLM and one to be handled by formal methods-based program synthesis.
  arXiv  Detail & Related papers  (2024-09-18T15:59:06Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [92.62952504133926]
  This study evaluated the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks.<n>We developed a taxonomy of bugs for incorrect codes and analyzed the root cause for common bug types.<n>We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• Where Is Self-admitted Code Generated by Large Language Models on GitHub? [14.13629953845785]
  This study investigates self-admitted code generated by Large Language Models on GitHub.<n>ChatGPT and Copilot dominate code generation, with minimal contributions from other LLMs.<n>Most code comments only state LLM use, while few include details like prompts, human edits, or code testing status.
  arXiv  Detail & Related papers  (2024-06-27T21:47:27Z)
• Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models [10.519984835232359]
  Large Language Models (LLMs) have demonstrated unprecedented capabilities in code generation.<n>We conducted an in-depth analysis of code generation errors across six representative LLMs on the HumanEval dataset.<n>Our findings highlighted several challenges in locating and fixing code generation errors made by LLMs.
  arXiv  Detail & Related papers  (2024-06-13T01:29:52Z)
• Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting [78.48355455324688]
  We propose a novel zero-shot synthetic code detector based on the similarity between the original code and its LLM-rewritten variants.<n>Our results demonstrate a significant improvement over existing SOTA synthetic content detectors.
  arXiv  Detail & Related papers  (2024-05-25T08:57:28Z)
• CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification [73.66920648926161]
  We introduce the concept of code hallucinations and propose a classification method for code hallucination based on execution verification.<n>We present a dynamic detection algorithm called CodeHalu designed to detect and quantify code hallucinations.<n>We also introduce the CodeHaluEval benchmark, which includes 8,883 samples from 699 tasks, to systematically and quantitatively evaluate code hallucinations.
  arXiv  Detail & Related papers  (2024-04-30T23:56:38Z)
• InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models [56.723509505549536]
  InfiBench is the first large-scale freeform question-answering (QA) benchmark for code to our knowledge. It comprises 234 carefully selected high-quality Stack Overflow questions that span across 15 programming languages. We conduct a systematic evaluation for over 100 latest code LLMs on InfiBench, leading to a series of novel and insightful findings.
  arXiv  Detail & Related papers  (2024-03-11T02:06:30Z)
• Assured LLM-Based Software Engineering [51.003878077888686]
  This paper is an outline of the content of the keynote by Mark Harman at the International Workshop on Interpretability, Robustness, and Benchmarking in Neural Software Engineering, Monday 15th April 2024, Lisbon, Portugal.
  arXiv  Detail & Related papers  (2024-02-06T20:38:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.