Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams

• URL: http://arxiv.org/abs/2411.05231v2
• Date: Thu, 12 Dec 2024 16:40:18 GMT
• Title: Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams
• Authors: Adriana Caraeni, Alexander Scarlatos, Andrew Lan,
• Abstract summary: We leverage state-of-the-art multi-modal AI models, in particular GPT-4o, to automatically grade handwritten responses to college-level math exams. Using real student responses to questions in a probability theory exam, we evaluate GPT-4o's alignment with ground-truth scores from human graders using various prompting techniques.
• Score: 48.99818550820575
• License:
• Abstract: Recent advances in generative artificial intelligence (AI) have shown promise in accurately grading open-ended student responses. However, few prior works have explored grading handwritten responses due to a lack of data and the challenge of combining visual and textual information. In this work, we leverage state-of-the-art multi-modal AI models, in particular GPT-4o, to automatically grade handwritten responses to college-level math exams. Using real student responses to questions in a probability theory exam, we evaluate GPT-4o's alignment with ground-truth scores from human graders using various prompting techniques. We find that while providing rubrics improves alignment, the model's overall accuracy is still too low for real-world settings, showing there is significant room for growth in this task.

Related papers

• Assessing instructor-AI cooperation for grading essay-type questions in an introductory sociology course [0.0]
  We evaluate generative pre-trained transformers (GPT) models' performance in transcribing and scoring students' responses. For grading, GPT demonstrated strong correlations with the human grader scores, especially when template answers were provided. This study contributes to the growing literature on AI in education, demonstrating its potential to enhance fairness and efficiency in grading essay-type questions.
  arXiv  Detail & Related papers  (2025-01-11T07:18:12Z)
• Can AI Assistance Aid in the Grading of Handwritten Answer Sheets? [2.025468874117372]
  This work introduces an AI-assisted grading pipeline. The pipeline first uses text detection to automatically detect question regions present in a question paper PDF. Next, it uses SOTA text detection methods to highlight important keywords present in the handwritten answer regions of scanned answer sheets to assist in the grading process.
  arXiv  Detail & Related papers  (2024-08-23T07:00:25Z)
• GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition? [82.40761196684524]
  This paper centers on the evaluation of GPT-4's linguistic and visual capabilities in zero-shot visual recognition tasks. We conduct extensive experiments to evaluate GPT-4's performance across images, videos, and point clouds. Our findings show that GPT-4, enhanced with rich linguistic descriptions, significantly improves zero-shot recognition.
  arXiv  Detail & Related papers  (2023-11-27T11:29:10Z)
• NERIF: GPT-4V for Automatic Scoring of Drawn Models [0.6278186810520364]
  Recently released GPT-4V provides a unique opportunity to advance scientific modeling practices. We developed a method employing instructional note and rubrics to prompt GPT-4V to score students' drawn models. GPT-4V scores were compared with human experts' scores to calculate scoring accuracy.
  arXiv  Detail & Related papers  (2023-11-21T20:52:04Z)
• GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks [70.98062518872999]
  We validate GPT-4V's capabilities for evaluation purposes, addressing tasks ranging from foundational image-to-text and text-to-image synthesis to high-level image-to-image translations and multi-images to text alignment. Notably, GPT-4V shows promising agreement with humans across various tasks and evaluation methods, demonstrating immense potential for multi-modal LLMs as evaluators.
  arXiv  Detail & Related papers  (2023-11-02T16:11:09Z)
• Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code [7.760653867600283]
  We evaluate GPT-4 using three prompt engineering strategies -- basic prompting, in-context learning, and task-specific prompting. We compare it against 17 fine-tuned models across three code-related tasks: code summarization, generation, and translation.
  arXiv  Detail & Related papers  (2023-10-11T00:21:00Z)
• Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation [25.317788211120362]
  We investigate the role of generative AI models in providing human tutor-style programming hints. Recent works have benchmarked state-of-the-art models for various feedback generation scenarios. We develop a novel technique, GPT4Hints-GPT3.5Val, to push the limits of generative AI models.
  arXiv  Detail & Related papers  (2023-10-05T17:02:59Z)
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts [170.01089233942594]
  MathVista is a benchmark designed to combine challenges from diverse mathematical and visual tasks. The best-performing GPT-4V model achieves an overall accuracy of 49.9%, substantially outperforming Bard, the second-best performer, by 15.1%. GPT-4V still falls short of human performance by 10.4%, as it often struggles to understand complex figures and perform rigorous reasoning.
  arXiv  Detail & Related papers  (2023-10-03T17:57:24Z)
• Is GPT-4 a Good Data Analyst? [67.35956981748699]
  We consider GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains. We design several task-specific evaluation metrics to systematically compare the performance between several professional human data analysts and GPT-4. Experimental results show that GPT-4 can achieve comparable performance to humans.
  arXiv  Detail & Related papers  (2023-05-24T11:26:59Z)
• Sparks of Artificial General Intelligence: Early experiments with GPT-4 [66.1188263570629]
  GPT-4, developed by OpenAI, was trained using an unprecedented scale of compute and data. We demonstrate that GPT-4 can solve novel and difficult tasks that span mathematics, coding, vision, medicine, law, psychology and more. We believe GPT-4 could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system.
  arXiv  Detail & Related papers  (2023-03-22T16:51:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.