Three-Phase Evaluation of AI-Assisted Software Development Life Cycle

• URL: http://arxiv.org/abs/2607.05125v1
• Date: Mon, 06 Jul 2026 14:13:25 GMT
• Title: Three-Phase Evaluation of AI-Assisted Software Development Life Cycle
• Abstract summary: A team of four developers reimplemented a full-stack web application using GitHub Copilot, an AI-exclusive workflow using GitHub Copilot, and an AI-exclusive workflow using AWS Kiro.<n>Across phases, higher levels of AI autonomy were associated with reduced development effort, improved requirement adherence, and lower self-reported mental workload.<n>The AWS Kiro phase achieved the strongest overall performance on most measured dimensions, suggesting that tooling architecture may influence outcomes independently of AI autonomy level.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper presents an exploratory evaluation of how increasing levels of AI autonomy affect software development productivity, requirement adherence, and developer cognitive workload. A team of four developers reimplemented the same full-stack web application across three sequential phases: partial AI-assisted development using GitHub Copilot, an AI-exclusive workflow using GitHub Copilot, and an AI-exclusive workflow using AWS Kiro. Evaluation metrics included development effort (hours), requirement adherence (RITM score), AI-interaction efficiency, and NASA-TLX workload measures. Across phases, higher levels of AI autonomy were associated with reduced development effort, improved requirement adherence, and lower self-reported mental workload, while developer frustration increased modestly. The AWS Kiro phase achieved the strongest overall performance on most measured dimensions, suggesting that tooling architecture may influence outcomes independently of AI autonomy level.

Related papers

• Practical Implementation Report on Introducing Spec-Driven Development Using AI Agents in Software Development PBL [1.7539923984360692]
  This study reports on the practical implementation of Spec-Driven Development, a software development methodology premised on AI agents.<n>We defined a workflow consisting of four phases, namely investigation, planning, implementation, and review.<n>We analyzed the results from three perspectives, namely students' subjective AI usage, implementation throughput, and code comprehension.
  arXiv  Detail & Related papers  (2026-08-31T10:47:29Z)
• You Shall Not Pass! Where and Why Developers Draw The Line on AI Autonomy [7.477031451733673]
  Most developers accepted AI producing work under their oversight, although accepted autonomy varied substantively across tasks and individuals.<n> Acceptance was lowest for identity-defining, human-facing, and design-oriented work, and higher among developers with more AI experience and risk tolerance.<n>Our findings suggest that preferences for AI autonomy reflect how developers cognitively experience their work.
  arXiv  Detail & Related papers  (2026-07-01T07:24:14Z)
• EmboCoach-Bench: Benchmarking AI Agents on Developing Embodied Robots [68.29056647487519]
  Embodied AI is fueled by high-fidelity simulation and large-scale data collection.<n>However, this scaling capability remains bottlenecked by a reliance on labor-intensive manual oversight.<n>We introduce textscEmboCoach-Bench, a benchmark evaluating the capacity of LLM agents to autonomously engineer embodied policies.
  arXiv  Detail & Related papers  (2026-01-29T11:33:49Z)
• AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development [12.50615284537175]
  Large language model (LLM) based coding agents increasingly act as autonomous contributors that generate and merge pull requests.<n>We present a longitudinal causal study of agent adoption in open-source repositories using staggered difference-in-differences with matched controls.
  arXiv  Detail & Related papers  (2026-01-20T04:51:56Z)
• SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience [71.82719117238307]
  We propose SEAgent, an agentic self-evolving framework enabling computer-use agents to evolve through interactions with unfamiliar software.<n>We validate the effectiveness of SEAgent across five novel software environments within OS-World.<n>Our approach achieves a significant improvement of 23.2% in success rate, from 11.3% to 34.5%, over a competitive open-source CUA.
  arXiv  Detail & Related papers  (2025-08-06T17:58:46Z)
• How Software Engineers Engage with AI: A Pragmatic Process Model and Decision Framework Grounded in Industry Observations [1.516251872371896]
  GitHub Copilot and ChatGPT have given rise to "vibe coding"<n>This paper presents two complementary contributions.<n>First, a pragmatic process model capturing real-world AI-assisted SE activities, including prompt design, inspection, fallback, and refinement.<n>Second, a 2D decision framework that could help developers reason about trade-offs between effort saved and output quality.
  arXiv  Detail & Related papers  (2025-07-23T21:00:21Z)
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [60.04362496037186]
  We present the first controlled study of developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants.<n>Our results show agents can assist developers in ways that surpass copilots.
  arXiv  Detail & Related papers  (2025-07-10T20:12:54Z)
• Challenges and Paths Towards AI for Software Engineering [55.95365538122656]
  We discuss progress in AI for software engineering in threefold manner.<n>First, we provide a structured taxonomy of concrete tasks in AI for software engineering.<n>Second, we outline several key bottlenecks that limit current approaches.
  arXiv  Detail & Related papers  (2025-03-28T17:17:57Z)
• Intelligent Mobile AI-Generated Content Services via Interactive Prompt Engineering and Dynamic Service Provisioning [55.641299901038316]
  AI-generated content can organize collaborative Mobile AIGC Service Providers (MASPs) at network edges to provide ubiquitous and customized content for resource-constrained users.<n>Such a paradigm faces two significant challenges: 1) raw prompts often lead to poor generation quality due to users' lack of experience with specific AIGC models, and 2) static service provisioning fails to efficiently utilize computational and communication resources.<n>We develop an interactive prompt engineering mechanism that leverages a Large Language Model (LLM) to generate customized prompt corpora and employs Inverse Reinforcement Learning (IRL) for policy imitation.
  arXiv  Detail & Related papers  (2025-02-17T03:05:20Z)
• AI's Impact on Traditional Software Development [0.0]
  The application of artificial intelligence (AI) has brought key shifts in conventional tactical software development.<n>This paper examines the technical aspect of integrating AI into prior traditional software development life cycle methodologies.
  arXiv  Detail & Related papers  (2025-02-05T14:58:09Z)
• TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks [55.03911355902567]
  We introduce TheAgentCompany, a benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker.<n>We find that the most competitive agent can complete 30% of tasks autonomously.<n>This paints a nuanced picture on task automation with simulating LM agents in a setting a real workplace.
  arXiv  Detail & Related papers  (2024-12-18T18:55:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.