Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes

• URL: http://arxiv.org/abs/2311.18452v1
• Date: Thu, 30 Nov 2023 10:52:28 GMT
• Title: Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes
• Abstract summary: This study focuses on the initial experiences of 62 participants who used a contextualized coding AI assistant -- named StackSpot AI -- in a controlled setting. Assistants' use resulted in significant time savings, easier access to documentation, and the generation of accurate codes for internal APIs. challenges associated with the knowledge sources necessary to make the coding assistant access more contextual information as well as variable responses and limitations in handling complex codes were observed.
• Score: 11.520721038793285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the rapidly advancing field of artificial intelligence, software development has emerged as a key area of innovation. Despite the plethora of general-purpose AI assistants available, their effectiveness diminishes in complex, domain-specific scenarios. Noting this limitation, both the academic community and industry players are relying on contextualized coding AI assistants. These assistants surpass general-purpose AI tools by integrating proprietary, domain-specific knowledge, offering precise and relevant solutions. Our study focuses on the initial experiences of 62 participants who used a contextualized coding AI assistant -- named StackSpot AI -- in a controlled setting. According to the participants, the assistants' use resulted in significant time savings, easier access to documentation, and the generation of accurate codes for internal APIs. However, challenges associated with the knowledge sources necessary to make the coding assistant access more contextual information as well as variable responses and limitations in handling complex codes were observed. The study's findings, detailing both the benefits and challenges of contextualized AI assistants, underscore their potential to revolutionize software development practices, while also highlighting areas for further refinement.

Related papers

• AgentForge: An Immersive Role-Playing Platform for Learning Agentic Software Engineering [4.609790123179083]
  We present textitAgentForge, an immersive learning system in which novices take on one of four software-engineering roles.<n>In each practice session, novices perform their chosen role while AI agents perform the remaining three.<n>In a study with 37 novice developers, participants achieved high task-completion rates with AI-agent support.
  arXiv  Detail & Related papers  (2026-08-04T18:58:18Z)
• From Conversation to Contribution: Characterizing Coding Agent in Open-Source Software [15.414964008939434]
  We study how developers' chat-based interactions with AI relate to subsequent open-source development and collaboration.<n>We find heavier AI use in smaller, less mature, and less collaborative repositories.<n>While most developers are willing to share their chat, concerns remain around appearing incompetent.
  arXiv  Detail & Related papers  (2026-07-06T22:44:45Z)
• Usage, Effects and Requirements for AI Coding Assistants in the Enterprise: An Empirical Study [4.01226690413941]
  We surveyed 57 developers about their experience with AI coding assistants and CodeLLMs.<n>We reviewed 35 user surveys on the usage, experience and expectations of professionals and students using AI coding assistants and CodeLLMs.
  arXiv  Detail & Related papers  (2026-01-27T22:57:20Z)
• "Can you feel the vibes?": An exploration of novice programmer engagement with vibe coding [42.82674998306379]
  "vibe coding" refers to creating software via natural language prompts rather than direct code authorship.<n>This paper reports on a one-day educational hackathon investigating how novice programmers and mixed-experience teams engage with vibe coding.
  arXiv  Detail & Related papers  (2025-12-02T13:32:23Z)
• Executable Knowledge Graphs for Replicating AI Research [65.41207324831583]
  Executable Knowledge Graphs (xKG) is a modular and pluggable knowledge base that automatically integrates technical insights, code snippets, and domain-specific knowledge extracted from scientific literature.<n>Code will released at https://github.com/zjunlp/xKG.
  arXiv  Detail & Related papers  (2025-10-20T17:53:23Z)
• Deploying AI for Signal Processing education: Selected challenges and intriguing opportunities [44.18936398140735]
  Article explores the use of AI tools to facilitate and enhance education.<n> Primers are provided on several core technical issues that arise when using AI in educational settings.<n>The article serves as a resource for researchers and educators seeking to advance AI's role in engineering education.
  arXiv  Detail & Related papers  (2025-09-10T19:19:26Z)
• "My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants [13.118506949442564]
  We identify 1,085 AI coding assistants from the Visual Studio Code Marketplace.<n>We then manually analyze the user reviews sampled from 32 AI coding assistants that have sufficient installations and reviews to construct a comprehensive taxonomy of user concerns and feedback about these assistants.<n>We propose five practical implications and suggestions to guide the enhancement of AI coding assistants that satisfy user needs.
  arXiv  Detail & Related papers  (2025-08-17T08:22:47Z)
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [66.1850490474361]
  We conduct the first academic study to explore developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants, GitHub Copilot and OpenHands.<n>Our results show agents have the potential to assist developers in ways that surpass copilots.
  arXiv  Detail & Related papers  (2025-07-10T20:12:54Z)
• Tool or Trouble? Exploring Student Attitudes Toward AI Coding Assistants [0.0]
  Students completed a programming task with access to AI support; in the second, they extended their solutions without AI.<n>Findings suggest that AI tools were perceived as helpful for understanding code and increasing confidence, particularly during initial development.<n>Students reported difficulties transferring knowledge to unaided tasks, revealing possible overreliance and gaps in conceptual understanding.
  arXiv  Detail & Related papers  (2025-06-26T05:59:23Z)
• Multi-agent Embodied AI: Advances and Future Directions [46.23631919950584]
  Embodied artificial intelligence (Embodied AI) plays a pivotal role in the application of advanced technologies in the intelligent era.<n>This paper reviews the current state of research, analyzes key contributions, and identifies challenges and future directions.
  arXiv  Detail & Related papers  (2025-05-08T10:13:53Z)
• Explainability for Embedding AI: Aspirations and Actuality [1.8130068086063336]
  Explainable AI (XAI) may allow developers to understand better the systems they build. Existing XAI systems still fall short of this aspiration. We see an unmet need to provide developers with adequate support mechanisms to cope with this complexity.
  arXiv  Detail & Related papers  (2025-04-20T14:20:01Z)
• The Impact of Generative AI Coding Assistants on Developers Who Are Visually Impaired [3.2895414694900684]
  We conducted a study where developers who are visually impaired completed a series of programming tasks using a generative AI coding assistant. While participants found the AI assistant beneficial and reported significant advantages, they also highlighted accessibility challenges. Our findings emphasize the need to apply activity-centered design principles to generative AI assistants.
  arXiv  Detail & Related papers  (2025-03-10T22:06:43Z)
• Human-AI Experience in Integrated Development Environments: A Systematic Literature Review [2.1749194587826026]
  In-IDE HAX explores the evolving dynamics of Human-Computer Interaction in AI-assisted coding environments. Our findings reveal that AI-assisted coding enhances developer productivity but also introduces challenges, such as verification overhead, automation bias, and over-reliance. Concerns about code correctness, security, and maintainability highlight the urgent need for explainability, verification mechanisms, and adaptive user control.
  arXiv  Detail & Related papers  (2025-03-08T12:40:18Z)
• CodeA11y: Making AI Coding Assistants Useful for Accessible Web Development [22.395216419337768]
  Despite specialized accessibility tools, novice developers often remain unaware of them, leading to 96% of web pages that contain accessibility violations. Our formative study with 16 developers revealed three key issues in AI-assisted coding: failure to prompt AI for accessibility, omitting crucial manual steps like replacing placeholder attributes, and the inability to verify compliance. To address these issues, we developed CodeA11y, a GitHub Copilot Extension, that suggests accessibility-compliant code and displays manual validation reminders.
  arXiv  Detail & Related papers  (2025-02-15T19:11:21Z)
• A Multi-Year Grey Literature Review on AI-assisted Test Automation [46.97326049485643]
  Test Automation (TA) techniques are crucial for quality assurance in software engineering but face limitations. Given the prevalent usage of AI in industry, sources of truth are held in grey literature as well as the minds of professionals. This study surveys grey literature to explore how AI is adopted in TA, focusing on the problems it solves, its solutions, and the available tools.
  arXiv  Detail & Related papers  (2024-08-12T15:26:36Z)
• Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward [9.177785129949]
  We aim to better understand how specifically developers are using AI assistants. We carried out a large-scale survey aimed at how AI assistants are used.
  arXiv  Detail & Related papers  (2024-06-11T23:10:43Z)
• The Ethics of Advanced AI Assistants [53.89899371095332]
  This paper focuses on the opportunities and the ethical and societal risks posed by advanced AI assistants. We define advanced AI assistants as artificial agents with natural language interfaces, whose function is to plan and execute sequences of actions on behalf of a user. We consider the deployment of advanced assistants at a societal scale, focusing on cooperation, equity and access, misinformation, economic impact, the environment and how best to evaluate advanced AI assistants.
  arXiv  Detail & Related papers  (2024-04-24T23:18:46Z)
• Envisioning the Next-Generation AI Coding Assistants: Insights & Proposals [5.641402231731082]
  We present four key takeaways from our experience developing in-IDE AI coding assistants. We propose open questions and challenges that academia and industry should address to realize the vision of next-generation AI coding assistants.
  arXiv  Detail & Related papers  (2024-03-21T17:47:28Z)
• Generating Java Methods: An Empirical Assessment of Four AI-Based Code Assistants [5.32539007352208]
  We assess the effectiveness of four popular AI-based code assistants, namely GitHub Copilot, Tabnine, ChatGPT, and Google Bard. Results show that Copilot is often more accurate than other techniques, yet none of the assistants is completely subsumed by the rest of the approaches.
  arXiv  Detail & Related papers  (2024-02-13T12:59:20Z)
• Toward enriched Cognitive Learning with XAI [44.99833362998488]
  We introduce an intelligent system (CL-XAI) for Cognitive Learning which is supported by artificial intelligence (AI) tools. The use of CL-XAI is illustrated with a game-inspired virtual use case where learners tackle problems to enhance problem-solving skills.
  arXiv  Detail & Related papers  (2023-12-19T16:13:47Z)
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review [9.355153561673855]
  This paper focuses on transformer-based large language models (LLMs) trained using Big Code. LLMs have played a crucial role in facilitating AI-assisted programming applications, including code generation, code completion, code translation, code refinement, code summarization, defect detection, and clone detection. It explores the challenges and opportunities associated with incorporating NLP techniques with software naturalness in these applications.
  arXiv  Detail & Related papers  (2023-07-04T21:26:51Z)
• AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges [60.56413461109281]
  Artificial Intelligence for IT operations (AIOps) aims to combine the power of AI with the big data generated by IT Operations processes. We discuss in depth the key types of data emitted by IT Operations activities, the scale and challenges in analyzing them, and where they can be helpful. We categorize the key AIOps tasks as - incident detection, failure prediction, root cause analysis and automated actions.
  arXiv  Detail & Related papers  (2023-04-10T15:38:12Z)
• OpenAGI: When LLM Meets Domain Experts [51.86179657467822]
  Human Intelligence (HI) excels at combining basic skills to solve complex tasks. This capability is vital for Artificial Intelligence (AI) and should be embedded in comprehensive AI Agents. We introduce OpenAGI, an open-source platform designed for solving multi-step, real-world tasks.
  arXiv  Detail & Related papers  (2023-04-10T03:55:35Z)
• Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
  We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
  arXiv  Detail & Related papers  (2023-02-14T18:43:34Z)
• Chatbots As Fluent Polyglots: Revisiting Breakthrough Code Snippets [0.0]
  The research applies AI-driven code assistants to analyze a selection of influential computer code that has shaped modern technology. The original contribution of this study was to examine half of the most significant code advances in the last 50 years.
  arXiv  Detail & Related papers  (2023-01-05T23:17:17Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)
• A User-Centred Framework for Explainable Artificial Intelligence in Human-Robot Interaction [70.11080854486953]
  We propose a user-centred framework for XAI that focuses on its social-interactive aspect. The framework aims to provide a structure for interactive XAI solutions thought for non-expert users.
  arXiv  Detail & Related papers  (2021-09-27T09:56:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.