When is Generated Code Difficult to Comprehend? Assessing AI Agent Python Code Proficiency in the Wild

• URL: http://arxiv.org/abs/2604.00299v1
• Date: Tue, 31 Mar 2026 22:49:44 GMT
• Title: When is Generated Code Difficult to Comprehend? Assessing AI Agent Python Code Proficiency in the Wild
• Abstract summary: This study investigates the code proficiency of AI agents to determine the skill level required for developers to maintain their code.<n>We mined 591 pull requests containing 5,027 Python files generated by three distinct AI agents.
• Score: 1.5432419759274403
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid adoption of AI coding agents is fundamentally shifting software developers' roles from code authors to code reviewers. While developers spend a significant portion of their time reading and comprehending code, the linguistic proficiency and complexity of the Python code generated by these agents remain largely unexplored. This study investigates the code proficiency of AI agents to determine the skill level required for developers to maintain their code. Leveraging the AIDev dataset, we mined 591 pull requests containing 5,027 Python files generated by three distinct AI agents and employed pycefr, a static analysis tool that maps Python constructs to six proficiency levels, ranging from A1 (Basic) to C2 (Mastery), to analyze the code. Our results reveal that: AI agents predominantly generate Basic-level code, with over 90% of constructs falling into the A1 and A2 categories, and less than 1% classified as Mastery (C2); AI agents' and humans' pull requests share a broadly similar proficiency profile; High-proficiency code by AI agents are from feature addition and bug fixing tasks. These findings suggest that while AI-generated code is generally accessible to developers with basic Python skills, specific tasks may require advanced proficiency to review and maintain complex, agent-generated constructs.

Related papers

• HybridCodeAuthorship: A Benchmark Dataset for Line-Level Code Authorship Detection [1.6385993554828697]
  HybridCodeAuthorship is a novel benchmark of Python code files with interleaved human- and AI-authored lines of code.<n>We then benchmark the performance of two state-of-the-art AI-generated code detection algorithms at both the line- and chunk-level.
  arXiv  Detail & Related papers  (2026-06-10T19:21:19Z)
• SWE-chat: Coding Agent Interactions From Real Users in the Wild [70.18158706281724]
  SWE-chat is the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild.<n>The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls.
  arXiv  Detail & Related papers  (2026-04-22T17:08:19Z)
• Mining Type Constructs Using Patterns in AI-Generated Code [1.2107297090229683]
  It remains unstudied whether AI essentially outperforms humans in type-related programming tasks.<n>We present the first empirical analysis to answer these questions in the domain of TypeScript projects.<n>Surprisingly, even with all these issues, Agentic pull requests have 1.8x higher acceptance rates compared to humans for TypeScript.
  arXiv  Detail & Related papers  (2026-02-20T03:17:42Z)
• AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios [49.90735676070039]
  The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow.<n>We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks.<n>We propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks.
  arXiv  Detail & Related papers  (2026-01-28T13:49:18Z)
• AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality [0.0]
  The rapid adoption of AI coding agents for software development has raised important questions about the quality and maintainability of the code they produce.<n>This data mining challenge focuses on AIDev, the first large-scale, openly available dataset capturing agent-pull requests from real-world GitHub repositories.<n>We identified 364 maintainability and security-related build smells across varying severity levels, indicating that AI-generated build code can introduce quality issues.
  arXiv  Detail & Related papers  (2026-01-23T15:40:28Z)
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [60.04362496037186]
  We present the first controlled study of developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants.<n>Our results show agents can assist developers in ways that surpass copilots.
  arXiv  Detail & Related papers  (2025-07-10T20:12:54Z)
• SciMaster: Towards General-Purpose Scientific AI Agents, Part I. X-Master as Foundation: Can We Lead on Humanity's Last Exam? [51.112225746095746]
  We introduce X-Master, a tool-augmented reasoning agent designed to emulate human researchers.<n>X-Masters sets a new state-of-the-art record on Humanity's Last Exam with a score of 32.1%.
  arXiv  Detail & Related papers  (2025-07-07T17:50:52Z)
• AIGCodeSet: A New Annotated Dataset for AI Generated Code Detection [0.0]
  We present AIGCodeSet, which consists of 2.828 AI-generated and 4.755 human-written Python codes.<n>Our experiments show that a Bayesian classifier outperforms the other models.
  arXiv  Detail & Related papers  (2024-12-21T11:53:49Z)
• The Rise and Potential of Large Language Model Based Agents: A Survey [91.71061158000953]
  Large language models (LLMs) are regarded as potential sparks for Artificial General Intelligence (AGI) We start by tracing the concept of agents from its philosophical origins to its development in AI, and explain why LLMs are suitable foundations for agents. We explore the extensive applications of LLM-based agents in three aspects: single-agent scenarios, multi-agent scenarios, and human-agent cooperation.
  arXiv  Detail & Related papers  (2023-09-14T17:12:03Z)
• Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
  We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
  arXiv  Detail & Related papers  (2023-02-14T18:43:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.