Automatically Generating Questions About Scratch Programs

• URL: http://arxiv.org/abs/2510.11658v1
• Date: Mon, 13 Oct 2025 17:34:12 GMT
• Title: Automatically Generating Questions About Scratch Programs
• Authors: Florian Obermüller, Gordon Fraser,
• Abstract summary: We propose a set of 30 different questions for Scratch code covering an established program comprehension model.<n>We extend the LitterBox static analysis tool to automatically generate corresponding questions for a given Scratch program.<n>On a dataset of 600,913 projects we generated 54,118,694 questions automatically.
• Score: 10.726203107125315
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When learning to program, students are usually assessed based on the code they wrote. However, the mere completion of a programming task does not guarantee actual comprehension of the underlying concepts. Asking learners questions about the code they wrote has therefore been proposed as a means to assess program comprehension. As creating targeted questions for individual student programs can be tedious and challenging, prior work has proposed to generate such questions automatically. In this paper we generalize this idea to the block-based programming language Scratch. We propose a set of 30 different questions for Scratch code covering an established program comprehension model, and extend the LitterBox static analysis tool to automatically generate corresponding questions for a given Scratch program. On a dataset of 600,913 projects we generated 54,118,694 questions automatically. Our initial experiments with 34 ninth graders demonstrate that this approach can indeed generate meaningful questions for Scratch programs, and we find that the ability of students to answer these questions on their programs relates to their overall performance.

Related papers

• Self-Questioning Language Models [58.73276539661649]
  We propose an asymmetric self-play framework where a proposer is given the topic and generates a question for a solver.<n>Both the proposer and solver are trained via reinforcement learning.<n>We study this asymmetric self-play framework on three benchmarks: three-digit multiplication, algebra problems from the OMEGA benchmark, and programming problems from Codeforces.
  arXiv  Detail & Related papers  (2025-08-05T17:51:33Z)
• Probing the Unknown: Exploring Student Interactions with Probeable Problems at Scale in Introductory Programming [4.1153199495993364]
  This study explores the use of Probeable Problems'', automatically gradable tasks that have deliberately vague or incomplete specifications.<n>Such problems require students to submit test inputs, or probes', to clarify requirements before implementation.<n> Systematic strategies, such as thoroughly exploring expected behavior before coding, resulted in fewer incorrect code submissions and correlated with course success.
  arXiv  Detail & Related papers  (2025-04-16T02:50:00Z)
• Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions [2.377308748205625]
  We explore the capability of the state-of-the-art LLMs in answering QLCs that are generated from code that the LLMs have created. Our results show that although the state-of-the-art LLMs can create programs and trace program execution when prompted, they easily succumb to similar errors that have previously been recorded for novice programmers.
  arXiv  Detail & Related papers  (2024-04-17T20:37:00Z)
• Automated Questions About Learners' Own Code Help to Detect Fragile Knowledge [0.0]
  Students are able to produce correctly functioning program code even though they have a fragile understanding of how it actually works. Questions derived automatically from individual exercise submissions (QLC) can probe if and how well the students understand the structure and logic of the code they just created.
  arXiv  Detail & Related papers  (2023-06-28T14:49:16Z)
• What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveys [63.51903260461746]
  We propose a novel task for knowledge-driven follow-up question generation in conversational surveys. We constructed a new human-annotated dataset of human-written follow-up questions with dialogue history and labeled knowledge. We then propose a two-staged knowledge-driven model for the task, which generates informative and coherent follow-up questions.
  arXiv  Detail & Related papers  (2022-05-23T00:57:33Z)
• CodeQA: A Question Answering Dataset for Source Code Comprehension [82.63394952538292]
  Given a code snippet and a question, a textual answer is required to be generated. CodeQA contains a Java dataset with 119,778 question-answer pairs and a Python dataset with 70,085 question-answer pairs.
  arXiv  Detail & Related papers  (2021-09-17T06:06:38Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)
• Students Struggle to Explain Their Own Program Code [0.0]
  We ask students to explain the structure and execution of their small programs after they submit them to a programming exercise. One third of the students struggled to explain their own program code. Our results indicate that answering properly aligned QLCs correctly has stronger correlation with student success and retention than merely submitting a correct program.
  arXiv  Detail & Related papers  (2021-04-14T09:13:05Z)
• Let's Ask Students About Their Programs, Automatically [0.0]
  Students sometimes produce code that works but that its author does not comprehend. One way to tackle these issues is to probe students' comprehension by asking them questions about their own programs. We propose an approach to automatically generate questions about student-written program code.
  arXiv  Detail & Related papers  (2021-03-20T09:15:37Z)
• Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning [56.771557756836906]
  We present a novel method that automatically learns a retrieval model alternately with the programmer from weak supervision. Our system leads to state-of-the-art performance on a large-scale task for complex question answering over knowledge bases.
  arXiv  Detail & Related papers  (2020-10-29T18:28:16Z)
• Inquisitive Question Generation for High Level Text Comprehension [60.21497846332531]
  We introduce INQUISITIVE, a dataset of 19K questions that are elicited while a person is reading through a document. We show that readers engage in a series of pragmatic strategies to seek information. We evaluate question generation models based on GPT-2 and show that our model is able to generate reasonable questions.
  arXiv  Detail & Related papers  (2020-10-04T19:03:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.