A comparison of Human, GPT-3.5, and GPT-4 Performance in a University-Level Coding Course

• URL: http://arxiv.org/abs/2403.16977v1
• Date: Mon, 25 Mar 2024 17:41:02 GMT
• Title: A comparison of Human, GPT-3.5, and GPT-4 Performance in a University-Level Coding Course
• Authors: Will Yeadon, Alex Peach, Craig P. Testrow,
• Abstract summary: This study evaluates the performance of ChatGPT variants, GPT-3.5 and GPT-4, against solely student work and a mixed category containing both student and GPT-4 contributions in university-level physics coding assignments using the Python language. Students averaged 91.9% (SE:0.4), surpassing the highest performing AI submission category, GPT-4 with prompt engineering, which scored 81.1% (SE:0.8) - a statistically significant difference (p = $2.482 times 10-10$) The blinded markers were tasked with guessing the authorship of the submissions on a four-point Likert scale from Definitely
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study evaluates the performance of ChatGPT variants, GPT-3.5 and GPT-4, both with and without prompt engineering, against solely student work and a mixed category containing both student and GPT-4 contributions in university-level physics coding assignments using the Python language. Comparing 50 student submissions to 50 AI-generated submissions across different categories, and marked blindly by three independent markers, we amassed $n = 300$ data points. Students averaged 91.9% (SE:0.4), surpassing the highest performing AI submission category, GPT-4 with prompt engineering, which scored 81.1% (SE:0.8) - a statistically significant difference (p = $2.482 \times 10^{-10}$). Prompt engineering significantly improved scores for both GPT-4 (p = $1.661 \times 10^{-4}$) and GPT-3.5 (p = $4.967 \times 10^{-9}$). Additionally, the blinded markers were tasked with guessing the authorship of the submissions on a four-point Likert scale from `Definitely AI' to `Definitely Human'. They accurately identified the authorship, with 92.1% of the work categorized as 'Definitely Human' being human-authored. Simplifying this to a binary `AI' or `Human' categorization resulted in an average accuracy rate of 85.3%. These findings suggest that while AI-generated work closely approaches the quality of university students' work, it often remains detectable by human evaluators.

Related papers

• Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams [48.99818550820575]
  We leverage state-of-the-art multi-modal AI models, in particular GPT-4o, to automatically grade handwritten responses to college-level math exams. Using real student responses to questions in a probability theory exam, we evaluate GPT-4o's alignment with ground-truth scores from human graders using various prompting techniques.
  arXiv  Detail & Related papers  (2024-11-07T22:51:47Z)
• Towards Scalable Automated Grading: Leveraging Large Language Models for Conceptual Question Evaluation in Engineering [5.160473221022088]
  This study explores the feasibility of using large language models (LLMs) for automated grading of conceptual questions. We compared the grading performance of GPT-4o with that of human teaching assistants (TAs) on ten quiz problems from the MEEN 361 course at Texas A&M University. Our analysis reveals that GPT-4o performs well when grading criteria are straightforward but struggles with nuanced answers.
  arXiv  Detail & Related papers  (2024-11-06T04:41:13Z)
• Whodunit: Classifying Code as Human Authored or GPT-4 Generated -- A case study on CodeChef problems [0.13124513975412253]
  We use code stylometry and machine learning to distinguish between GPT-4 generated and human-authored code. Our dataset comprises human-authored solutions from CodeChef and AI-authored solutions generated by GPT-4. Our study shows that code stylometry is a promising approach for distinguishing between GPT-4 generated code and human-authored code.
  arXiv  Detail & Related papers  (2024-03-06T19:51:26Z)
• If in a Crowdsourced Data Annotation Pipeline, a GPT-4 [12.898580978312848]
  This paper compared GPT-4 and an ethical and well-executed MTurk pipeline. Despite best practices, MTurk pipeline's highest accuracy was 81.5%, whereas GPT-4 achieved 83.6%.
  arXiv  Detail & Related papers  (2024-02-26T18:08:52Z)
• GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation [93.55550787058012]
  This paper presents an automatic, versatile, and human-aligned evaluation metric for text-to-3D generative models. To this end, we first develop a prompt generator using GPT-4V to generate evaluating prompts. We then design a method instructing GPT-4V to compare two 3D assets according to user-defined criteria.
  arXiv  Detail & Related papers  (2024-01-08T18:52:09Z)
• GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition? [82.40761196684524]
  This paper centers on the evaluation of GPT-4's linguistic and visual capabilities in zero-shot visual recognition tasks. We conduct extensive experiments to evaluate GPT-4's performance across images, videos, and point clouds. Our findings show that GPT-4, enhanced with rich linguistic descriptions, significantly improves zero-shot recognition.
  arXiv  Detail & Related papers  (2023-11-27T11:29:10Z)
• GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions [22.181665641802468]
  This paper investigates using large language models (LLMs) as a cost-effective, reference-free method for evaluating figure captions. We first constructed SCICAP-EVAL, a human evaluation dataset that contains human judgments for 3,600 scientific figure captions. We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption based on its potential to aid reader understanding.
  arXiv  Detail & Related papers  (2023-10-23T23:24:57Z)
• Fine-tuning ChatGPT for Automatic Scoring [1.4833692070415454]
  This study highlights the potential of fine-tuned ChatGPT (GPT3.5) for automatically scoring student written constructed responses. We compare the performance of fine-tuned GPT-3.5 with the fine-tuned state-of-the-art Google's generated language model, BERT.
  arXiv  Detail & Related papers  (2023-10-16T05:09:16Z)
• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models [66.12432440863816]
  We propose Prometheus, a fully open-source Large Language Model (LLM) that is on par with GPT-4's evaluation capabilities. Prometheus scores a Pearson correlation of 0.897 with human evaluators when evaluating with 45 customized score rubrics. Prometheus achieves the highest accuracy on two human preference benchmarks.
  arXiv  Detail & Related papers  (2023-10-12T16:50:08Z)
• Progressive-Hint Prompting Improves Reasoning in Large Language Models [63.98629132836499]
  This paper proposes a new prompting method, named Progressive-Hint Prompting (PHP) It enables automatic multiple interactions between users and Large Language Models (LLMs) by using previously generated answers as hints to progressively guide toward the correct answers. We conducted extensive and comprehensive experiments on seven benchmarks. The results show that PHP significantly improves accuracy while remaining highly efficient.
  arXiv  Detail & Related papers  (2023-04-19T16:29:48Z)
• GPT-4 Technical Report [116.90398195245983]
  GPT-4 is a large-scale, multimodal model which can accept image and text inputs and produce text outputs. It exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers.
  arXiv  Detail & Related papers  (2023-03-15T17:15:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.