Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes

• URL: http://arxiv.org/abs/2311.18452v1
• Date: Thu, 30 Nov 2023 10:52:28 GMT
• Title: Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes
• Authors: Gustavo Pinto and Cleidson de Souza and Thayssa Rocha and Igor Steinmacher and Alberto de Souza and Edward Monteiro
• Abstract summary: This study focuses on the initial experiences of 62 participants who used a contextualized coding AI assistant -- named StackSpot AI -- in a controlled setting. Assistants' use resulted in significant time savings, easier access to documentation, and the generation of accurate codes for internal APIs. challenges associated with the knowledge sources necessary to make the coding assistant access more contextual information as well as variable responses and limitations in handling complex codes were observed.
• Score: 11.520721038793285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the rapidly advancing field of artificial intelligence, software development has emerged as a key area of innovation. Despite the plethora of general-purpose AI assistants available, their effectiveness diminishes in complex, domain-specific scenarios. Noting this limitation, both the academic community and industry players are relying on contextualized coding AI assistants. These assistants surpass general-purpose AI tools by integrating proprietary, domain-specific knowledge, offering precise and relevant solutions. Our study focuses on the initial experiences of 62 participants who used a contextualized coding AI assistant -- named StackSpot AI -- in a controlled setting. According to the participants, the assistants' use resulted in significant time savings, easier access to documentation, and the generation of accurate codes for internal APIs. However, challenges associated with the knowledge sources necessary to make the coding assistant access more contextual information as well as variable responses and limitations in handling complex codes were observed. The study's findings, detailing both the benefits and challenges of contextualized AI assistants, underscore their potential to revolutionize software development practices, while also highlighting areas for further refinement.

Related papers

• Explainability for Embedding AI: Aspirations and Actuality [1.8130068086063336]
  Explainable AI (XAI) may allow developers to understand better the systems they build. Existing XAI systems still fall short of this aspiration. We see an unmet need to provide developers with adequate support mechanisms to cope with this complexity.
  arXiv  Detail & Related papers  (2025-04-20T14:20:01Z)
• The Impact of Generative AI Coding Assistants on Developers Who Are Visually Impaired [3.2895414694900684]
  We conducted a study where developers who are visually impaired completed a series of programming tasks using a generative AI coding assistant. While participants found the AI assistant beneficial and reported significant advantages, they also highlighted accessibility challenges. Our findings emphasize the need to apply activity-centered design principles to generative AI assistants.
  arXiv  Detail & Related papers  (2025-03-10T22:06:43Z)
• Human-AI Experience in Integrated Development Environments: A Systematic Literature Review [2.1749194587826026]
  In-IDE HAX explores the evolving dynamics of Human-Computer Interaction in AI-assisted coding environments. Our findings reveal that AI-assisted coding enhances developer productivity but also introduces challenges, such as verification overhead, automation bias, and over-reliance. Concerns about code correctness, security, and maintainability highlight the urgent need for explainability, verification mechanisms, and adaptive user control.
  arXiv  Detail & Related papers  (2025-03-08T12:40:18Z)
• CodeA11y: Making AI Coding Assistants Useful for Accessible Web Development [22.395216419337768]
  Despite specialized accessibility tools, novice developers often remain unaware of them, leading to 96% of web pages that contain accessibility violations. Our formative study with 16 developers revealed three key issues in AI-assisted coding: failure to prompt AI for accessibility, omitting crucial manual steps like replacing placeholder attributes, and the inability to verify compliance. To address these issues, we developed CodeA11y, a GitHub Copilot Extension, that suggests accessibility-compliant code and displays manual validation reminders.
  arXiv  Detail & Related papers  (2025-02-15T19:11:21Z)
• A Multi-Year Grey Literature Review on AI-assisted Test Automation [46.97326049485643]
  Test Automation (TA) techniques are crucial for quality assurance in software engineering but face limitations. Given the prevalent usage of AI in industry, sources of truth are held in grey literature as well as the minds of professionals. This study surveys grey literature to explore how AI is adopted in TA, focusing on the problems it solves, its solutions, and the available tools.
  arXiv  Detail & Related papers  (2024-08-12T15:26:36Z)
• Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward [9.177785129949]
  We aim to better understand how specifically developers are using AI assistants. We carried out a large-scale survey aimed at how AI assistants are used.
  arXiv  Detail & Related papers  (2024-06-11T23:10:43Z)
• The Ethics of Advanced AI Assistants [53.89899371095332]
  This paper focuses on the opportunities and the ethical and societal risks posed by advanced AI assistants. We define advanced AI assistants as artificial agents with natural language interfaces, whose function is to plan and execute sequences of actions on behalf of a user. We consider the deployment of advanced assistants at a societal scale, focusing on cooperation, equity and access, misinformation, economic impact, the environment and how best to evaluate advanced AI assistants.
  arXiv  Detail & Related papers  (2024-04-24T23:18:46Z)
• Envisioning the Next-Generation AI Coding Assistants: Insights & Proposals [5.641402231731082]
  We present four key takeaways from our experience developing in-IDE AI coding assistants. We propose open questions and challenges that academia and industry should address to realize the vision of next-generation AI coding assistants.
  arXiv  Detail & Related papers  (2024-03-21T17:47:28Z)
• Generating Java Methods: An Empirical Assessment of Four AI-Based Code Assistants [5.32539007352208]
  We assess the effectiveness of four popular AI-based code assistants, namely GitHub Copilot, Tabnine, ChatGPT, and Google Bard. Results show that Copilot is often more accurate than other techniques, yet none of the assistants is completely subsumed by the rest of the approaches.
  arXiv  Detail & Related papers  (2024-02-13T12:59:20Z)
• Toward enriched Cognitive Learning with XAI [44.99833362998488]
  We introduce an intelligent system (CL-XAI) for Cognitive Learning which is supported by artificial intelligence (AI) tools. The use of CL-XAI is illustrated with a game-inspired virtual use case where learners tackle problems to enhance problem-solving skills.
  arXiv  Detail & Related papers  (2023-12-19T16:13:47Z)
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review [9.355153561673855]
  This paper focuses on transformer-based large language models (LLMs) trained using Big Code. LLMs have played a crucial role in facilitating AI-assisted programming applications, including code generation, code completion, code translation, code refinement, code summarization, defect detection, and clone detection. It explores the challenges and opportunities associated with incorporating NLP techniques with software naturalness in these applications.
  arXiv  Detail & Related papers  (2023-07-04T21:26:51Z)
• AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges [60.56413461109281]
  Artificial Intelligence for IT operations (AIOps) aims to combine the power of AI with the big data generated by IT Operations processes. We discuss in depth the key types of data emitted by IT Operations activities, the scale and challenges in analyzing them, and where they can be helpful. We categorize the key AIOps tasks as - incident detection, failure prediction, root cause analysis and automated actions.
  arXiv  Detail & Related papers  (2023-04-10T15:38:12Z)
• OpenAGI: When LLM Meets Domain Experts [51.86179657467822]
  Human Intelligence (HI) excels at combining basic skills to solve complex tasks. This capability is vital for Artificial Intelligence (AI) and should be embedded in comprehensive AI Agents. We introduce OpenAGI, an open-source platform designed for solving multi-step, real-world tasks.
  arXiv  Detail & Related papers  (2023-04-10T03:55:35Z)
• Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
  We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
  arXiv  Detail & Related papers  (2023-02-14T18:43:34Z)
• Chatbots As Fluent Polyglots: Revisiting Breakthrough Code Snippets [0.0]
  The research applies AI-driven code assistants to analyze a selection of influential computer code that has shaped modern technology. The original contribution of this study was to examine half of the most significant code advances in the last 50 years.
  arXiv  Detail & Related papers  (2023-01-05T23:17:17Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)
• A User-Centred Framework for Explainable Artificial Intelligence in Human-Robot Interaction [70.11080854486953]
  We propose a user-centred framework for XAI that focuses on its social-interactive aspect. The framework aims to provide a structure for interactive XAI solutions thought for non-expert users.
  arXiv  Detail & Related papers  (2021-09-27T09:56:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.