The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers

• URL: http://arxiv.org/abs/2405.17739v1
• Date: Tue, 28 May 2024 01:48:28 GMT
• Title: The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
• Authors: James Prather, Brent Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S. Randrianasolo, Brett Becker, Bailey Kimmel, Jared Wright, Ben Briggs,
• Abstract summary: Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Many novices are now programming with generative AI (GenAI) Our findings show an unfortunate divide in the use of GenAI tools between students who accelerated and students who struggled.
• Score: 1.995977018536036
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who accelerated and students who struggled. Students who accelerated were able to use GenAI to create code they already intended to make and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.

Related papers

• Misconceptions, Pragmatism, and Value Tensions: Evaluating Students' Understanding and Perception of Generative AI for Education [0.11704154007740832]
  Students are early adopters of the technology, utilizing it in atypical ways. Students were asked to describe 1) their understanding of GenAI; 2) their use of GenAI; 3) their opinions on the benefits, downsides, and ethical issues pertaining to its use in education.
  arXiv  Detail & Related papers  (2024-10-29T17:41:06Z)
• Model-based Maintenance and Evolution with GenAI: A Look into the Future [47.93555901495955]
  We argue that Generative Artificial Intelligence (GenAI) can be used as a means to address the limitations of Model-Based Engineering (MBM&E) We propose that GenAI can be used in MBM&E for: reducing engineers' learning curve, maximizing efficiency with recommendations, or serving as a reasoning tool to understand domain problems.
  arXiv  Detail & Related papers  (2024-07-09T23:13:26Z)
• Probeable Problems for Beginner-level Programming-with-AI Contests [0.0]
  We conduct a 2-hour programming contest for undergraduate Computer Science students from multiple institutions. Students were permitted to work individually or in groups, and were free to use AI tools. We analyze the extent to which the code submitted by these groups identifies missing details and identify ways in which Probeable Problems can support learning in formal and informal CS educational contexts.
  arXiv  Detail & Related papers  (2024-05-24T00:39:32Z)
• Genetic Auto-prompt Learning for Pre-trained Code Intelligence Language Models [54.58108387797138]
  We investigate the effectiveness of prompt learning in code intelligence tasks. Existing automatic prompt design methods are very limited to code intelligence tasks. We propose Genetic Auto Prompt (GenAP) which utilizes an elaborate genetic algorithm to automatically design prompts.
  arXiv  Detail & Related papers  (2024-03-20T13:37:00Z)
• Transforming Software Development with Generative AI: Empirical Insights on Collaboration and Workflow [2.6124032579630114]
  Generative AI (GenAI) has fundamentally changed how knowledge workers, such as software developers, solve tasks and collaborate to build software products. Introducing innovative tools like ChatGPT and Copilot has created new opportunities to assist and augment software developers across various problems. Our study reveals that ChatGPT signifies a paradigm shift in the workflow of software developers. The technology empowers developers by enabling them to work more efficiently, speed up the learning process, and increase motivation by reducing tedious and repetitive tasks.
  arXiv  Detail & Related papers  (2024-02-12T12:36:29Z)
• Identifying and Mitigating the Security Risks of Generative AI [179.2384121957896]
  This paper reports the findings of a workshop held at Google on the dual-use dilemma posed by GenAI. GenAI can be used just as well by attackers to generate new attacks and increase the velocity and efficacy of existing attacks. We discuss short-term and long-term goals for the community on this topic.
  arXiv  Detail & Related papers  (2023-08-28T18:51:09Z)
• Innovating Computer Programming Pedagogy: The AI-Lab Framework for Generative AI Adoption [0.0]
  We introduce "AI-Lab," a framework for guiding students in effectively leveraging GenAI within core programming courses. By identifying and rectifying GenAI's errors, students enrich their learning process. For educators, AI-Lab provides mechanisms to explore students' perceptions of GenAI's role in their learning experience.
  arXiv  Detail & Related papers  (2023-08-23T17:20:37Z)
• Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
  We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
  arXiv  Detail & Related papers  (2023-02-14T18:43:34Z)
• Seamful XAI: Operationalizing Seamful Design in Explainable AI [59.89011292395202]
  Mistakes in AI systems are inevitable, arising from both technical limitations and sociotechnical gaps. We propose that seamful design can foster AI explainability by revealing sociotechnical and infrastructural mismatches. We explore this process with 43 AI practitioners and real end-users.
  arXiv  Detail & Related papers  (2022-11-12T21:54:05Z)
• A User-Centred Framework for Explainable Artificial Intelligence in Human-Robot Interaction [70.11080854486953]
  We propose a user-centred framework for XAI that focuses on its social-interactive aspect. The framework aims to provide a structure for interactive XAI solutions thought for non-expert users.
  arXiv  Detail & Related papers  (2021-09-27T09:56:23Z)
• Steps Before Syntax: Helping Novice Programmers Solve Problems using the PCDIT Framework [2.768397481213625]
  Novice programmers often struggle with problem solving due to the high cognitive loads they face. Many introductory programming courses do not explicitly teach it, assuming that problem solving skills are acquired along the way. We present 'PCDIT', a non-linear problem solving framework that provides scaffolding to guide novice programmers through the process of transforming a problem specification into an implemented and tested solution for an imperative programming language.
  arXiv  Detail & Related papers  (2021-09-18T10:31:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.