Artificial-Intelligence Grading Assistance for Handwritten Components of a Calculus Exam

• URL: http://arxiv.org/abs/2510.05162v1
• Date: Sat, 04 Oct 2025 15:07:06 GMT
• Title: Artificial-Intelligence Grading Assistance for Handwritten Components of a Calculus Exam
• Abstract summary: In a large first-year exam, students' handwritten work was graded by GPT-5 against the same rubric used by teaching assistants (TAs)<n>We calibrated a human-in-the-loop filter that combines a partial-credit threshold with an Item Response Theory (2PL) risk measure based on the deviation between the AI score and the model-expected score for each student-item.<n>Unfiltered AI-TA agreement was moderate, adequate for low-stakes feedback but not for high-stakes use.
• Score: 41.99844472131922
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate whether contemporary multimodal LLMs can assist with grading open-ended calculus at scale without eroding validity. In a large first-year exam, students' handwritten work was graded by GPT-5 against the same rubric used by teaching assistants (TAs), with fractional credit permitted; TA rubric decisions served as ground truth. We calibrated a human-in-the-loop filter that combines a partial-credit threshold with an Item Response Theory (2PL) risk measure based on the deviation between the AI score and the model-expected score for each student-item. Unfiltered AI-TA agreement was moderate, adequate for low-stakes feedback but not for high-stakes use. Confidence filtering made the workload-quality trade-off explicit: under stricter settings, AI delivered human-level accuracy, but also left roughly 70% of the items to be graded by humans. Psychometric patterns were constrained by low stakes on the open-ended portion, a small set of rubric checkpoints, and occasional misalignment between designated answer regions and where work appeared. Practical adjustments such as slightly higher weight and protected time, a few rubric-visible substeps, stronger spatial anchoring should raise ceiling performance. Overall, calibrated confidence and conservative routing enable AI to reliably handle a sizable subset of routine cases while reserving expert judgment for ambiguous or pedagogically rich responses.

Related papers

• Automated Grading of Handwritten Mathematics Using Vision-Capable LLMs [2.727845763576068]
  Vision-capable large language models (LLMs) offer new opportunities.<n>We present an empirical evaluation of an LLM-based grader for handwritten mathematical work using instructor-defined rubrics.
  arXiv  Detail & Related papers  (2026-05-18T19:06:15Z)
• Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation [0.3078691410268859]
  Automated short answer scoring (ASAS) is shifting from discriminative, fine-tuned models to large language models (LLMs) used in few-shot settings.<n>We investigate the relationship between the degree of task-specific adaptation of different models and quality-conditioned scoring agreement.
  arXiv  Detail & Related papers  (2026-05-08T12:12:01Z)
• Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement [50.34437999083224]
  Autoraters are increasingly used for evaluation and automated moderation content.<n>There is limited statistical analysis of how modifications in a rubric presented to both humans and autoraters affect their score agreement.
  arXiv  Detail & Related papers  (2026-05-07T13:55:11Z)
• Designing Reliable LLM-Assisted Rubric Scoring for Constructed Responses: Evidence from Physics Exams [1.6955250288861923]
  Student responses in STEM assessments are often handwritten and combine symbolic expressions, calculations, and diagrams.<n>Recent advances in large language models (LLMs) have increased attention to AI-assisted scoring.<n>This study examined the reliability of AI-assisted scoring of undergraduate physics constructed responses using GPT-4o.
  arXiv  Detail & Related papers  (2026-04-14T03:04:44Z)
• LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias [3.9562034587217187]
  We analyze agreement with human consensus scores, directional bias, and the stability of bias estimates.<n>We observe large and stable negative directional bias on Lower-Order Concern (LOC) traits, such as Grammar and Conventions.<n>This analysis shows that LOC bias is often detectable with very small validation sets.
  arXiv  Detail & Related papers  (2026-03-31T21:40:36Z)
• Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark [9.922581736690159]
  We present a large-scale empirical study of AI grading on real, handwritten calculus work from UC Irvine.<n>Using OCR-conditioned large language models with structured, rubric-guided prompting, our system produces scores and formative feedback for thousands of free-response quiz submissions.<n>In a setting with no single ground-truth label, we evaluate performance against official teaching-assistant grades, student surveys, and independent human review.
  arXiv  Detail & Related papers  (2026-03-01T03:32:51Z)
• Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness [4.129847064263056]
  We systematically evaluate the performance of Large Language Models for rubric-based short-answer grading.<n>We find that alignment is strong for binary tasks but degrades with increased rubric granularity.<n>Experiments reveal that while the model is resilient to prompt injection, it is sensitive to synonym substitutions.
  arXiv  Detail & Related papers  (2025-12-21T05:22:04Z)
• Evaluating Generative AI for CS1 Code Grading: Direct vs Reverse Methods [0.0]
  This paper compares two AI-based grading techniques: textitDirect, where the AI model applies a rubric directly to student code, and textitReverse (a newly proposed approach), where the AI first fixes errors, then deduces a grade based on the nature and number of fixes.<n>We discuss the strengths and limitations of each approach, practical considerations for prompt design, and future directions for hybrid human-AI grading systems.
  arXiv  Detail & Related papers  (2025-11-17T01:38:06Z)
• Evaluating the Promise and Pitfalls of LLMs in Hiring Decisions [1.1883838320818292]
  Large language models (LLMs) in hiring promise to streamline candidate screening, but it also raises serious concerns regarding accuracy and algorithmic bias.<n>We benchmark several state-of-the-art foundational LLMs and compare them with our proprietary domain-specific hiring model (Match Score) for job candidate matching.<n>Our experiments show that Match Score outperforms the general-purpose LLMs on accuracy (ROC AUC 0.85 vs 0.77) and achieves significantly more equitable outcomes across demographic groups.
  arXiv  Detail & Related papers  (2025-07-02T19:02:18Z)
• T2I-Eval-R1: Reinforcement Learning-Driven Reasoning for Interpretable Text-to-Image Evaluation [60.620408007636016]
  We propose T2I-Eval-R1, a novel reinforcement learning framework that trains open-source MLLMs using only coarse-grained quality scores.<n>Our approach integrates Group Relative Policy Optimization into the instruction-tuning process, enabling models to generate both scalar scores and interpretable reasoning chains.
  arXiv  Detail & Related papers  (2025-05-23T13:44:59Z)
• ChatGPT for automated grading of short answer questions in mechanical ventilation [0.0]
  Large language models (LLMs) simulate conversational language and interpret unstructured free-text responses.<n>We evaluated ChatGPT 4o to grade SAQs in a postgraduate medical setting using data from 215 students.
  arXiv  Detail & Related papers  (2025-05-05T19:04:25Z)
• PredictaBoard: Benchmarking LLM Score Predictability [50.47497036981544]
  Large Language Models (LLMs) often fail unpredictably.<n>This poses a significant challenge to ensuring their safe deployment.<n>We present PredictaBoard, a novel collaborative benchmarking framework.
  arXiv  Detail & Related papers  (2025-02-20T10:52:38Z)
• Probably Approximately Precision and Recall Learning [60.00180898830079]
  A key challenge in machine learning is the prevalence of one-sided feedback.<n>We introduce a Probably Approximately Correct (PAC) framework in which hypotheses are set functions that map each input to a set of labels.<n>We develop new algorithms that learn from positive data alone, achieving optimal sample complexity in the realizable case.
  arXiv  Detail & Related papers  (2024-11-20T04:21:07Z)
• Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing [37.92922713921964]
  Persona-driven role-playing (PRP) aims to build AI characters that can respond to user queries by faithfully sticking with all persona statements. This paper presents a pioneering exploration to quantify PRP faithfulness as a fine-grained and explainable criterion, which also serves as a reliable reference for optimization.
  arXiv  Detail & Related papers  (2024-05-13T13:21:35Z)
• Making Large Language Models Better Reasoners with Alignment [57.82176656663245]
  Reasoning is a cognitive process of using evidence to reach a sound conclusion. Recent studies reveal that fine-tuning LLMs on data with the chain of thought (COT) reasoning process can significantly enhance their reasoning capabilities. We introduce an textitAlignment Fine-Tuning (AFT) paradigm, which involves three steps.
  arXiv  Detail & Related papers  (2023-09-05T11:32:48Z)
• Distractor generation for multiple-choice questions with predictive prompting and large language models [21.233186754403093]
  Large Language Models (LLMs) such as ChatGPT have demonstrated remarkable performance across various tasks. We propose a strategy for guiding LLMs in generating relevant distractors by prompting them with question items automatically retrieved from a question bank. We found that on average 53% of the generated distractors presented to the teachers were rated as high-quality, i.e., suitable for immediate use as is.
  arXiv  Detail & Related papers  (2023-07-30T23:15:28Z)
• Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels [92.98756432746482]
  We study a weakly supervised problem called learning with complementary labels. We show that the quality of gradient estimation matters more in risk minimization. We propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance.
  arXiv  Detail & Related papers  (2020-07-05T04:19:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.