Navigating Compiler Errors with AI Assistance - A Study of GPT Hints in an Introductory Programming Course

• URL: http://arxiv.org/abs/2403.12737v1
• Date: Tue, 19 Mar 2024 13:54:14 GMT
• Title: Navigating Compiler Errors with AI Assistance - A Study of GPT Hints in an Introductory Programming Course
• Authors: Maciej Pankiewicz, Ryan S. Baker,
• Abstract summary: We examined the efficacy of AI-assisted learning in an introductory programming course at the university level. We used a GPT-4 model to generate personalized hints for compiler errors within a platform for automated assessment of programming assignments. For the six most commonly occurring error types we observed mixed results in terms of performance when access to GPT hints was enabled for the experimental group.
• Score: 0.23020018305241333
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We examined the efficacy of AI-assisted learning in an introductory programming course at the university level by using a GPT-4 model to generate personalized hints for compiler errors within a platform for automated assessment of programming assignments. The control group had no access to GPT hints. In the experimental condition GPT hints were provided when a compiler error was detected, for the first half of the problems in each module. For the latter half of the module, hints were disabled. Students highly rated the usefulness of GPT hints. In affect surveys, the experimental group reported significantly higher levels of focus and lower levels of confrustion (confusion and frustration) than the control group. For the six most commonly occurring error types we observed mixed results in terms of performance when access to GPT hints was enabled for the experimental group. However, in the absence of GPT hints, the experimental group's performance surpassed the control group for five out of the six error types.

Related papers

• Constrained C-Test Generation via Mixed-Integer Programming [55.28927994487036]
  This work proposes a novel method to generate C-Tests; a form of cloze tests (a gap filling exercise) where only the last part of a word is turned into a gap. In contrast to previous works that only consider varying the gap size or gap placement to achieve locally optimal solutions, we propose a mixed-integer programming (MIP) approach. We publish our code, model, and collected data consisting of 32 English C-Tests with 20 gaps each (totaling 3,200 individual gap responses) under an open source license.
  arXiv  Detail & Related papers  (2024-04-12T21:35:21Z)
• Feedback-Generation for Programming Exercises With GPT-4 [0.0]
  This paper explores the quality of GPT-4 Turbo's generated output for prompts containing both the programming task specification and a student's submission as input. The output was qualitatively analyzed regarding correctness, personalization, fault localization, and other features identified in the material.
  arXiv  Detail & Related papers  (2024-03-07T12:37:52Z)
• Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks [53.936643052339]
  We evaluate the reasoning abilities of text-only and multimodal versions of GPT-4. Our experimental results support the conclusion that neither version of GPT-4 has developed robust abstraction abilities at humanlike levels.
  arXiv  Detail & Related papers  (2023-11-14T04:33:49Z)
• AI-enhanced Auto-correction of Programming Exercises: How Effective is GPT-3.5? [0.0]
  This paper investigates the potential of AI in providing personalized code correction and generating feedback. GPT-3.5 exhibited weaknesses in its evaluation, including localization of errors that were not the actual errors, or even hallucinated errors.
  arXiv  Detail & Related papers  (2023-10-24T10:35:36Z)
• Large Language Models (GPT) for automating feedback on programming assignments [0.0]
  We employ OpenAI's GPT-3.5 model to generate personalized hints for students solving programming assignments. Students rated the usefulness of GPT-generated hints positively.
  arXiv  Detail & Related papers  (2023-06-30T21:57:40Z)
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models [92.6951708781736]
  This work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5. We find that GPT models can be easily misled to generate toxic and biased outputs and leak private information. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps.
  arXiv  Detail & Related papers  (2023-06-20T17:24:23Z)
• Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses [0.0]
  GPT models evolved from completely failing the typical programming class' assessments to confidently passing the courses with no human involvement. This study provides evidence that programming instructors need to prepare for a world in which there is an easy-to-use technology that can be utilized by learners to collect passing scores.
  arXiv  Detail & Related papers  (2023-06-15T22:12:34Z)
• Generalized Planning in PDDL Domains with Pretrained Large Language Models [82.24479434984426]
  We consider PDDL domains and use GPT-4 to synthesize Python programs. We evaluate this approach in seven PDDL domains and compare it to four ablations and four baselines.
  arXiv  Detail & Related papers  (2023-05-18T14:48:20Z)
• Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction [28.58384091374763]
  GPT-3 and GPT-4 models are powerful, achieving high performance on a variety of Natural Language Processing tasks. We perform experiments testing the capabilities of a GPT-3.5 model (text-davinci-003) and a GPT-4 model (gpt-4-0314) on major GEC benchmarks. We report the performance of our best prompt on the BEA-2019 and JFLEG datasets, finding that the GPT models can perform well in a sentence-level revision setting.
  arXiv  Detail & Related papers  (2023-03-25T03:08:49Z)
• AGRO: Adversarial Discovery of Error-prone groups for Robust Optimization [109.91265884632239]
  Group distributionally robust optimization (G-DRO) can minimize the worst-case loss over a set of pre-defined groups over training data. We propose AGRO -- Adversarial Group discovery for Distributionally Robust Optimization. AGRO results in 8% higher model performance on average on known worst-groups, compared to prior group discovery approaches.
  arXiv  Detail & Related papers  (2022-12-02T00:57:03Z)
• Outlier-Robust Group Inference via Gradient Space Clustering [50.87474101594732]
  Existing methods can improve the worst-group performance, but they require group annotations, which are often expensive and sometimes infeasible to obtain. We address the problem of learning group annotations in the presence of outliers by clustering the data in the space of gradients of the model parameters. We show that data in the gradient space has a simpler structure while preserving information about minority groups and outliers, making it suitable for standard clustering methods like DBSCAN.
  arXiv  Detail & Related papers  (2022-10-13T06:04:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.