Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses

• URL: http://arxiv.org/abs/2310.15317v1
• Date: Mon, 23 Oct 2023 19:35:01 GMT
• Title: Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses
• Authors: Aysa Xuemo Fan, Ranran Haoran Zhang, Luc Paquette, Rui Zhang
• Abstract summary: Large language models (LLMs) can be used to generate code-tracing questions in programming courses. We present a dataset of human and LLM-generated tracing questions, serving as a valuable resource for both the education and NLP research communities.
• Score: 6.43363776610849
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we explore the application of large language models (LLMs) for generating code-tracing questions in introductory programming courses. We designed targeted prompts for GPT4, guiding it to generate code-tracing questions based on code snippets and descriptions. We established a set of human evaluation metrics to assess the quality of questions produced by the model compared to those created by human experts. Our analysis provides insights into the capabilities and potential of LLMs in generating diverse code-tracing questions. Additionally, we present a unique dataset of human and LLM-generated tracing questions, serving as a valuable resource for both the education and NLP research communities. This work contributes to the ongoing dialogue on the potential uses of LLMs in educational settings.

Related papers

• SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models [54.78329741186446]
  We propose a novel paradigm that uses a code-based critic model to guide steps including question-code data construction, quality control, and complementary evaluation. Experiments across both in-domain and out-of-domain benchmarks in English and Chinese demonstrate the effectiveness of the proposed paradigm.
  arXiv  Detail & Related papers  (2024-08-28T06:33:03Z)
• Systematic Task Exploration with LLMs: A Study in Citation Text Generation [63.50597360948099]
  Large language models (LLMs) bring unprecedented flexibility in defining and executing complex, creative natural language generation (NLG) tasks. We propose a three-component research framework that consists of systematic input manipulation, reference data, and output measurement. We use this framework to explore citation text generation -- a popular scholarly NLP task that lacks consensus on the task definition and evaluation metric.
  arXiv  Detail & Related papers  (2024-07-04T16:41:08Z)
• Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applications [0.4857223913212445]
  In the era of generative artificial intelligence (AI), the fusion of large language models (LLMs) offers unprecedented opportunities for innovation in the field of modern education. We investigate the effectiveness of prompt-based techniques in generating open-ended questions from school-level textbooks, assess their efficiency in generating open-ended questions from undergraduate-level technical textbooks, and explore the feasibility of employing a chain-of-thought inspired multi-stage prompting approach for language-agnostic multiple-choice question (MCQ) generation.
  arXiv  Detail & Related papers  (2024-05-19T15:13:51Z)
• Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions [2.377308748205625]
  We explore the capability of the state-of-the-art LLMs in answering QLCs that are generated from code that the LLMs have created. Our results show that although the state-of-the-art LLMs can create programs and trace program execution when prompted, they easily succumb to similar errors that have previously been recorded for novice programmers.
  arXiv  Detail & Related papers  (2024-04-17T20:37:00Z)
• Large Language Models for Data Annotation: A Survey [49.8318827245266]
  The emergence of advanced Large Language Models (LLMs) presents an unprecedented opportunity to automate the complicated process of data annotation. This survey includes an in-depth taxonomy of data types that LLMs can annotate, a review of learning strategies for models utilizing LLM-generated annotations, and a detailed discussion of the primary challenges and limitations associated with using LLMs for data annotation.
  arXiv  Detail & Related papers  (2024-02-21T00:44:04Z)
• Next-Step Hint Generation for Introductory Programming Using Large Language Models [0.8002196839441036]
  Large Language Models possess skills such as answering questions, writing essays or solving programming exercises. This work explores how LLMs can contribute to programming education by supporting students with automated next-step hints.
  arXiv  Detail & Related papers  (2023-12-03T17:51:07Z)
• A Large Language Model Approach to Educational Survey Feedback Analysis [0.0]
  This paper assesses the potential for the large language models (LLMs) GPT-4 and GPT-3.5 to aid in deriving insight from education feedback surveys.
  arXiv  Detail & Related papers  (2023-09-29T17:57:23Z)
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models [102.00201523306986]
  We present L2CEval, a systematic evaluation of the language-to-code generation capabilities of large language models (LLMs) We analyze the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition to assessing model performance, we measure confidence calibration for the models and conduct human evaluations of the output programs.
  arXiv  Detail & Related papers  (2023-09-29T17:57:00Z)
• Aligning Large Language Models with Human: A Survey [53.6014921995006]
  Large Language Models (LLMs) trained on extensive textual corpora have emerged as leading solutions for a broad array of Natural Language Processing (NLP) tasks. Despite their notable performance, these models are prone to certain limitations such as misunderstanding human instructions, generating potentially biased content, or factually incorrect information. This survey presents a comprehensive overview of these alignment technologies, including the following aspects.
  arXiv  Detail & Related papers  (2023-07-24T17:44:58Z)
• Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding [45.5690960017762]
  This study explores the use of large language models (LLMs) in supporting deductive coding. Instead of training task-specific models, a pre-trained LLM could be used directly for various tasks without fine-tuning through prompt learning. Using a curiosity-driven questions coding task as a case study, we found, by combining GPT-3 with expert-drafted codebooks, our proposed approach achieved fair to substantial agreements with expert-coded results.
  arXiv  Detail & Related papers  (2023-04-17T04:52:43Z)
• What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveys [63.51903260461746]
  We propose a novel task for knowledge-driven follow-up question generation in conversational surveys. We constructed a new human-annotated dataset of human-written follow-up questions with dialogue history and labeled knowledge. We then propose a two-staged knowledge-driven model for the task, which generates informative and coherent follow-up questions.
  arXiv  Detail & Related papers  (2022-05-23T00:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.