Probing the Unknown: Exploring Student Interactions with Probeable Problems at Scale in Introductory Programming

• URL: http://arxiv.org/abs/2504.11723v1
• Date: Wed, 16 Apr 2025 02:50:00 GMT
• Title: Probing the Unknown: Exploring Student Interactions with Probeable Problems at Scale in Introductory Programming
• Authors: Paul Denny, Viraj Kumar, Stephen MacNeil, James Prather, Juho Leinonen,
• Abstract summary: This study explores the use of Probeable Problems'', automatically gradable tasks that have deliberately vague or incomplete specifications.<n>Such problems require students to submit test inputs, or probes', to clarify requirements before implementation.<n> Systematic strategies, such as thoroughly exploring expected behavior before coding, resulted in fewer incorrect code submissions and correlated with course success.
• Score: 4.1153199495993364
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Introductory programming courses often rely on small code-writing exercises that have clearly specified problem statements. This limits opportunities for students to practice how to clarify ambiguous requirements -- a critical skill in real-world programming. In addition, the emerging capabilities of large language models (LLMs) to produce code from well-defined specifications may harm student engagement with traditional programming exercises. This study explores the use of ``Probeable Problems'', automatically gradable tasks that have deliberately vague or incomplete specifications. Such problems require students to submit test inputs, or `probes', to clarify requirements before implementation. Through analysis of over 40,000 probes in an introductory course, we identify patterns linking probing behaviors to task success. Systematic strategies, such as thoroughly exploring expected behavior before coding, resulted in fewer incorrect code submissions and correlated with course success. Feedback from nearly 1,000 participants highlighted the challenges and real-world relevance of these tasks, as well as benefits to critical thinking and metacognitive skills. Probeable Problems are easy to set up and deploy at scale, and help students recognize and resolve uncertainties in programming problems.

Related papers

• From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions [9.032718302451501]
  We introduce Prompt2Constraints, a novel method that translates students prompts into logical constraints. We use this approach to analyze a dataset of 1,872 prompts from 203 students solving programming tasks. We find that while successful and unsuccessful attempts tend to use a similar number of constraints overall, when students fail, they often modify their prompts more significantly.
  arXiv  Detail & Related papers  (2025-04-25T20:58:16Z)
• Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critique [66.94905631175209]
  We propose a novel inference-time scaling approach -- stepwise natural language self-critique (PANEL)<n>It employs self-generated natural language critiques as feedback to guide the step-level search process.<n>This approach bypasses the need for task-specific verifiers and the associated training overhead.
  arXiv  Detail & Related papers  (2025-03-21T17:59:55Z)
• Knowledge Tracing in Programming Education Integrating Students' Questions [0.0]
  This paper introduces SQKT (Students' Question-based Knowledge Tracing), a knowledge tracing model that leverages students' questions and automatically extracted skill information.<n> Experimental results demonstrate SQKT's superior performance in predicting student completion across various Python programming courses of differing difficulty levels.<n> SQKT can be used to tailor educational content to individual learning needs and design adaptive learning systems in computer science education.
  arXiv  Detail & Related papers  (2025-01-22T14:13:40Z)
• Integrating Natural Language Prompting Tasks in Introductory Programming Courses [3.907735250728617]
  This report explores the inclusion of two prompt-focused activities in an introductory programming course. The first requires students to solve computational problems by writing natural language prompts, emphasizing problem-solving over syntax. The second involves students crafting prompts to generate code equivalent to provided fragments, to foster an understanding of the relationship between prompts and code.
  arXiv  Detail & Related papers  (2024-10-04T01:03:25Z)
• Estimating Difficulty Levels of Programming Problems with Pre-trained Model [18.92661958433282]
  The difficulty level of each programming problem serves as an essential reference for guiding students' adaptive learning. We formulate the problem of automatic difficulty level estimation of each programming problem, given its textual description and a solution example of code. For tackling this problem, we propose to couple two pre-trained models, one for text modality and the other for code modality, into a unified model.
  arXiv  Detail & Related papers  (2024-06-13T05:38:20Z)
• Creating a Trajectory for Code Writing: Algorithmic Reasoning Tasks [0.923607423080658]
  This paper describes instruments and the machine learning models used for validating them. We have used the data collected in an introductory programming course in the penultimate week of the semester. Preliminary research suggests ART type instruments can be combined with specific machine learning models to act as an effective learning trajectory.
  arXiv  Detail & Related papers  (2024-04-03T05:07:01Z)
• Large Language Models as Analogical Reasoners [155.9617224350088]
  Chain-of-thought (CoT) prompting for language models demonstrates impressive performance across reasoning tasks. We introduce a new prompting approach, analogical prompting, designed to automatically guide the reasoning process of large language models.
  arXiv  Detail & Related papers  (2023-10-03T00:57:26Z)
• Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
  Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. Standard approaches require instructors to manually grade student-implemented interactive programs. Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
  arXiv  Detail & Related papers  (2022-11-16T10:00:23Z)
• Competition-Level Code Generation with AlphaCode [74.87216298566942]
  We introduce AlphaCode, a system for code generation that can create novel solutions to problems that require deeper reasoning. In simulated evaluations on recent programming competitions on the Codeforces platform, AlphaCode achieved on average a ranking of top 54.3%.
  arXiv  Detail & Related papers  (2022-02-08T23:16:31Z)
• Steps Before Syntax: Helping Novice Programmers Solve Problems using the PCDIT Framework [2.768397481213625]
  Novice programmers often struggle with problem solving due to the high cognitive loads they face. Many introductory programming courses do not explicitly teach it, assuming that problem solving skills are acquired along the way. We present 'PCDIT', a non-linear problem solving framework that provides scaffolding to guide novice programmers through the process of transforming a problem specification into an implemented and tested solution for an imperative programming language.
  arXiv  Detail & Related papers  (2021-09-18T10:31:15Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)
• Measuring Coding Challenge Competence With APPS [54.22600767666257]
  We introduce APPS, a benchmark for code generation. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. Recent models such as GPT-Neo can pass approximately 15% of the test cases of introductory problems.
  arXiv  Detail & Related papers  (2021-05-20T17:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.