LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability

• URL: http://arxiv.org/abs/2607.01247v1
• Date: Mon, 01 Jun 2026 18:00:34 GMT
• Title: LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability
• Abstract summary: Open-ended mathematics exams are valuable because they assess reasoning, proof construction, algorithmic thinking, and communication of intermediate steps.<n>This paper evaluates six contemporary large language model (LLM) configurations as grading assistants for an undergraduate discrete mathematics examination.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-ended mathematics exams are valuable because they assess reasoning, proof construction, algorithmic thinking, and communication of intermediate steps. They are also difficult to grade at scale because instructors must apply partial-credit rubrics consistently while giving feedback that helps students repair misconceptions. This paper evaluates six contemporary large language model (LLM) configurations, Gemini 3.1 Pro Extended, Gemini 3.5 Flash, ChatGPT 5.5 Pro Extended, ChatGPT 5.5 Thinking, Claude Pro Opus 4.7, and Claude Sonnet 4.6, as grading assistants for an undergraduate discrete mathematics examination. The study compares two grading policies. The BASELINE policy uses a stricter rubric-following prompt that emphasizes explicit evidence and complete justification. The LIBERAL policy was added after preliminary grading showed that the baseline condition sometimes applied harsh point deductions and failed to recognize valid partial reasoning. Agreement with human grading is measured at both the question and exam-total levels using mean absolute error, root mean squared error, normalized root mean squared error, Pearson correlation, and exact agreement. The results show that liberal partial-credit prompting reduces average question-level error for every evaluated model family. ChatGPT 5.5 Thinking (LIBERAL) has the lowest average question-level MAE (1.87) and RMSE (2.53), while Gemini 3.1 Pro Extended (LIBERAL) has the lowest total-score MAE (8.00) and RMSE (10.66). However, the strongest total-score Pearson correlation occurs under Gemini 3.1 Pro Extended (BASELINE) at 0.58, showing that point calibration and rank preservation remain distinct goals. We also report practical usability observations.

Related papers

• Vision-Language Models for Criterion-Level Grading of Handwritten Examinations in Outcome-Based Education [0.3686530147760243]
  This study evaluates vision-language models for handwritten outcome-based assessment.<n>We compare 20 configurations spanning Qwen2.5-VL, InternVL3, Pixtral, a Donut baseline, and a cascade ensemble.
  arXiv  Detail & Related papers  (2026-09-13T04:55:55Z)
• Can LLMs in Draft-Verify-Revise Pipelines Resolve Deictic Ambiguity? [0.0]
  Draft-verify-revise is an orchestration pattern for scaling inference-time compute.<n>Draft-verify-revise pipelines should be wary of deictic shifts and make the intended referent explicit at each stage.
  arXiv  Detail & Related papers  (2026-09-10T19:49:06Z)
• SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning [54.402550664505576]
  On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models to short-context students.<n>We study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models.<n> Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning.
  arXiv  Detail & Related papers  (2026-08-14T12:57:02Z)
• GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory [0.0134610399747818]
  GTBench is a curriculum-grounded benchmark for evaluating large language models (LLMs) as mathematical research assistants in graph theory.<n>We evaluate GPT-5, Claude Sonnet 4.6, Gemini 2.5 Flash-Lite, Llama 3.3 70B, and Mistral Large 3 under zero-shot and chain-of-thought prompting.
  arXiv  Detail & Related papers  (2026-06-02T04:40:25Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches [61.30693283718321]
  We present LiveMathematicianBench, a dynamic multiple-choice benchmark for research-level mathematical reasoning.<n>By grounding evaluation in newly published theorems, it provides a realistic testbed beyond memorized patterns.<n>It employs a proof-sketchguided distractor pipeline that uses high-level proof strategies to construct plausible but invalid answer choices.
  arXiv  Detail & Related papers  (2026-04-02T08:22:17Z)
• Implicit Grading Bias in Large Language Models: How Writing Style Affects Automated Assessment Across Math, Programming, and Essay Tasks [0.0]
  This study investigates whether large language models (LLMs) exhibit implicit grading bias based on writing style when the underlying content correctness remains constant.<n>Two state-of-the-art open-source LLMs were prompted to grade responses on a 1-10 scale with explicit instructions to evaluate content correctness only and to disregard writing style.<n>Our results reveal statistically significant grading bias in Essay/Writing tasks across both models and all perturbation types.
  arXiv  Detail & Related papers  (2026-03-19T11:20:07Z)
• QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs [29.26861081722613]
  We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic Alignment Gap when applied to upper-undergraduate to early graduate level mathematics.<n>We introduce QEDBench, the first large-scale dual-rubric alignment benchmark to measure alignment with human experts on university-level math.<n>We reveal that certain frontier evaluators like Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick exhibit significant positive bias.
  arXiv  Detail & Related papers  (2026-02-24T07:23:28Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• ChatGPT for automated grading of short answer questions in mechanical ventilation [0.0]
  Large language models (LLMs) simulate conversational language and interpret unstructured free-text responses.<n>We evaluated ChatGPT 4o to grade SAQs in a postgraduate medical setting using data from 215 students.
  arXiv  Detail & Related papers  (2025-05-05T19:04:25Z)
• Evaluating Mathematical Reasoning Beyond Accuracy [50.09931172314218]
  We introduce ReasonEval, a new methodology for evaluating the quality of reasoning steps.<n>We show that ReasonEval consistently outperforms baseline methods in the meta-evaluation datasets.<n>We observe that ReasonEval can play a significant role in data selection.
  arXiv  Detail & Related papers  (2024-04-08T17:18:04Z)
• Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings [63.35165397320137]
  This study investigates the consistency of feedback ratings generated by OpenAI's GPT-4. The model rated responses to tasks within the Higher Education subject domain of macroeconomics in terms of their content and style.
  arXiv  Detail & Related papers  (2023-08-03T12:47:17Z)
• BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics [70.52570641514146]
  We present a benchmark of unfaithful minimal pairs (BUMP) BUMP is a dataset of 889 human-written, minimally different summary pairs. Unlike non-pair-based datasets, BUMP can be used to measure the consistency of metrics.
  arXiv  Detail & Related papers  (2022-12-20T02:17:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.