How Do Software Professionals Evaluate AI-Generated Code? (Registered Report)

• URL: http://arxiv.org/abs/2607.09434v1
• Date: Fri, 10 Jul 2026 14:01:00 GMT
• Title: How Do Software Professionals Evaluate AI-Generated Code? (Registered Report)
• Abstract summary: Recent advances in generative AI tools have significantly changed how software professionals write, evaluate, and interact with code.<n>Despite the growing adoption of and reliance on these tools, it remains unclear as to how software professionals evaluate the code they generate.<n>This research aims to build a theory of how software professionals evaluate AI-generated code, grounded in their accounts of evaluative practices, perceptions, and preferences.
• Score: 4.3772632092306045
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in generative AI tools have significantly changed how software professionals write, evaluate, and interact with code. Generative AI tools such as GitHub Copilot, ChatGPT, and Claude are increasingly being integrated into everyday workflows. Despite the growing adoption of and reliance on these tools, it remains unclear as to how software professionals evaluate the code they generate. To explore this topic, we will conduct a constructivist grounded theory study that incorporates a survey, semi-structured interviews, and laddering interviews. With the initial survey data collection complete, we aim to interview 20--50 software professionals iteratively until theoretical saturation is achieved. This research aims to build a theory of how software professionals evaluate AI-generated code, grounded in their accounts of evaluative practices, perceptions, and preferences.

Related papers

• How Developers Discuss Generative AI: A Longitudinal Study of the Visual Studio Code Community [0.0]
  Generative AI tools such as GitHub Copilot, ChatGPT, and coding agents have rapidly become part of everyday software development.<n>This paper presents a longitudinal analysis of generative-AI-related discussions in the Visual Studio Code (VS Code) GitHub repository.
  arXiv  Detail & Related papers  (2026-09-04T03:16:08Z)
• Empirical Study on the Characteristics and Evolution of AI-usage in GitHub Repositories: Evidence from Code Comments [7.213400161126317]
  We analyze 35,361 GitHub code comments that explicitly reference AI use and their associated code blocks.<n>We first open-code 500 unique comments and code blocks to derive a taxonomy of AI-assisted development activities.<n>We then annotate the full dataset using two LLM-based classifiers and aggregate predictions with Dawid-Skene expectation-maximization.
  arXiv  Detail & Related papers  (2026-06-05T02:37:44Z)
• AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
  We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
  arXiv  Detail & Related papers  (2026-05-18T17:08:26Z)
• Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review [2.2807488730605296]
  We present a vision for an AI-powered code review workflow combining specialized agents with human-controlled quality.<n>We identify major open challenges for responsible adoption, including reliability, bias, privacy, automation bias, transparency, and evaluation.
  arXiv  Detail & Related papers  (2026-05-17T17:04:21Z)
• AI Co-Scientist for Ranking: Discovering Novel Search Ranking Models alongside LLM-based AI Agents with Cloud Computing Access [60.37646376282544]
  We present an AI Co-Scientist framework that automates the full search ranking research pipeline.<n>This is the first study in the ranking community to utilize an AI Co-Scientist framework for algorithmic research.
  arXiv  Detail & Related papers  (2026-03-23T09:29:08Z)
• Building Software by Rolling the Dice: A Qualitative Study of Vibe Coding [15.145249560710377]
  "vibe coders" build software primarily through prompts rather than writing code.<n>We conducted a theory study of 20 vibe-coding videos, including 7 live-streamed coding sessions and 13 opinion videos.<n>Our findings reveal a spectrum of behaviors: some vibe coders rely almost entirely on AI, while others examine and adapt generated outputs.
  arXiv  Detail & Related papers  (2025-12-27T00:38:37Z)
• From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence [150.3696990310269]
  Large language models (LLMs) have transformed automated software development by enabling direct translation of natural language descriptions into functional code.<n>We provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs.<n>We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder)
  arXiv  Detail & Related papers  (2025-11-23T17:09:34Z)
• Prompting in Practice: Investigating Software Developers' Use of Generative AI Tools [17.926187565860232]
  The integration of generative artificial intelligence (GenAI) tools has fundamentally transformed software development.<n>This study presents a systematic investigation of how software engineers integrate GenAI tools into their professional practice.<n>We surveyed 91 software engineers, including 72 active GenAI users, to understand AI usage patterns throughout the development process.
  arXiv  Detail & Related papers  (2025-10-07T15:02:22Z)
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [60.04362496037186]
  We present the first controlled study of developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants.<n>Our results show agents can assist developers in ways that surpass copilots.
  arXiv  Detail & Related papers  (2025-07-10T20:12:54Z)
• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)
• Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
  We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
  arXiv  Detail & Related papers  (2023-02-14T18:43:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.