Understanding Student Perceptions, Mistakes, and Debugging Approaches when Solving Natural Language Programming Tasks

• URL: http://arxiv.org/abs/2607.05034v1
• Date: Mon, 06 Jul 2026 13:16:39 GMT
• Title: Understanding Student Perceptions, Mistakes, and Debugging Approaches when Solving Natural Language Programming Tasks
• Abstract summary: We studied attempts by more than 900 students to solve dialogue-based Prompt Problems.<n>Compared to traditional coding tasks, students found prompting easier, more enjoyable, and better targeted at developing problem-solving skills.<n>The most common mistakes are related to the omission of key details, suggesting both a failure to acknowledge their importance and over-reliance on AI to infer them.
• Score: 18.517538299978465
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning to communicate with code-generating AI models is an emerging skill for novice programmers. One recent pedagogical approach, Prompt Problems, has students solve computational tasks by writing natural-language prompts for code-generating AI models. However, little is known about the specific prompt-level mistakes novice programmers make, the kinds of computational details they fail to communicate, and what strategies they use to recover when generated code is incorrect. In a CS1 course, we studied attempts by more than 900 students to solve dialogue-based Prompt Problems. We analyzed student reflections, unsuccessful prompts, and reported debugging strategies. Compared to traditional coding tasks, students generally found prompting easier, more enjoyable, and better targeted at developing problem-solving skills. The most common mistakes are related to the omission of key details, suggesting both a failure to acknowledge their importance and over-reliance on AI to infer them. When prompts failed, students focused more on clarifying their intent and reflecting on the provided problem details than on tracing generated code or examining test cases.

Related papers

• When AI Is Wrong on Purpose: How Students Respond to Buggy GenAI Code [18.653714524727153]
  We investigate how prompt-centered programming activities can be adapted to better foster these practices.<n>Specifically, we explore an approach where realistic, runnable bugs are injected into otherwise correct solutions.
  arXiv  Detail & Related papers  (2026-07-06T13:36:05Z)
• Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math [55.83696908107408]
  We introduce ScratchMath, a novel benchmark for explaining and classifying errors in authentic handwritten mathematics scratchwork.<n>Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students.<n>We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts.
  arXiv  Detail & Related papers  (2026-03-26T02:57:20Z)
• Probing the Unknown: Exploring Student Interactions with Probeable Problems at Scale in Introductory Programming [4.1153199495993364]
  This study explores the use of Probeable Problems'', automatically gradable tasks that have deliberately vague or incomplete specifications.<n>Such problems require students to submit test inputs, or probes', to clarify requirements before implementation.<n> Systematic strategies, such as thoroughly exploring expected behavior before coding, resulted in fewer incorrect code submissions and correlated with course success.
  arXiv  Detail & Related papers  (2025-04-16T02:50:00Z)
• Integrating Natural Language Prompting Tasks in Introductory Programming Courses [3.907735250728617]
  This report explores the inclusion of two prompt-focused activities in an introductory programming course. The first requires students to solve computational problems by writing natural language prompts, emphasizing problem-solving over syntax. The second involves students crafting prompts to generate code equivalent to provided fragments, to foster an understanding of the relationship between prompts and code.
  arXiv  Detail & Related papers  (2024-10-04T01:03:25Z)
• A Knowledge-Component-Based Methodology for Evaluating AI Assistants [9.412070852474313]
  We evaluate an automatic hint generator for CS1 programming assignments powered by GPT-4. This system provides natural language guidance about how students can improve their incorrect solutions to short programming exercises.
  arXiv  Detail & Related papers  (2024-06-09T00:58:39Z)
• Genetic Auto-prompt Learning for Pre-trained Code Intelligence Language Models [54.58108387797138]
  We investigate the effectiveness of prompt learning in code intelligence tasks. Existing automatic prompt design methods are very limited to code intelligence tasks. We propose Genetic Auto Prompt (GenAP) which utilizes an elaborate genetic algorithm to automatically design prompts.
  arXiv  Detail & Related papers  (2024-03-20T13:37:00Z)
• Large Language Models as Analogical Reasoners [155.9617224350088]
  Chain-of-thought (CoT) prompting for language models demonstrates impressive performance across reasoning tasks. We introduce a new prompting approach, analogical prompting, designed to automatically guide the reasoning process of large language models.
  arXiv  Detail & Related papers  (2023-10-03T00:57:26Z)
• Dcc --help: Generating Context-Aware Compiler Error Explanations with Large Language Models [53.04357141450459]
  dcc --help was deployed to our CS1 and CS2 courses, with 2,565 students using the tool over 64,000 times in ten weeks. We found that the LLM-generated explanations were conceptually accurate in 90% of compile-time and 75% of run-time cases, but often disregarded the instruction not to provide solutions in code.
  arXiv  Detail & Related papers  (2023-08-23T02:36:19Z)
• Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators [5.458849730200646]
  This paper introduces a novel pedagogical concept known as a Prompt Problem' A Prompt Problem challenges a student to create a natural language prompt that leads an LLM to produce the correct code for a specific problem. We report empirical findings from a field study in which Promptly was deployed in a first-year Python programming course.
  arXiv  Detail & Related papers  (2023-07-31T01:46:42Z)
• Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
  Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. Standard approaches require instructors to manually grade student-implemented interactive programs. Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
  arXiv  Detail & Related papers  (2022-11-16T10:00:23Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.