Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math

• URL: http://arxiv.org/abs/2603.24961v1
• Date: Thu, 26 Mar 2026 02:57:20 GMT
• Title: Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math
• Abstract summary: We introduce ScratchMath, a novel benchmark for explaining and classifying errors in authentic handwritten mathematics scratchwork.<n>Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students.<n>We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts.
• Score: 55.83696908107408
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Assessing student handwritten scratchwork is crucial for personalized educational feedback but presents unique challenges due to diverse handwriting, complex layouts, and varied problem-solving approaches. Existing educational NLP primarily focuses on textual responses and neglects the complexity and multimodality inherent in authentic handwritten scratchwork. Current multimodal large language models (MLLMs) excel at visual reasoning but typically adopt an "examinee perspective", prioritizing generating correct answers rather than diagnosing student errors. To bridge these gaps, we introduce ScratchMath, a novel benchmark specifically designed for explaining and classifying errors in authentic handwritten mathematics scratchwork. Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students, supporting two key tasks: Error Cause Explanation (ECE) and Error Cause Classification (ECC), with seven defined error types. The dataset is meticulously annotated through rigorous human-machine collaborative approaches involving multiple stages of expert labeling, review, and verification. We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts, especially in visual recognition and logical reasoning. Proprietary models notably outperform open-source models, with large reasoning models showing strong potential for error explanation. All evaluation data and frameworks are publicly available to facilitate further research.

Related papers

• Finding Common Mistakes In Modelling With Mathematical Formalisms Using LLMs [0.0]
  We present a tool-supported workflow that allows to identify candidates for common mistakes that explain many student mistakes in large educational data sets.<n>The visualization is designed to help researchers to identify common modelling mistakes.<n>We show that this approach works well by reproducing common mistakes in propositional logic modelling that were identified by hand in the literature.
  arXiv  Detail & Related papers  (2026-09-15T12:40:27Z)
• Simulating Students' Java Programming Errors with Large Language Models [7.543318372822572]
  This paper explores whether large language models (LLMs) can serve as scalable proxies for students by simulating realistic logical errors in code submissions.<n>Using the CodeWorkout dataset of 74,000+ unique student Java submissions across 37 problems, we evaluate five LLMs under three mainstream prompting strategies.<n>Our quantitative findings reveal that while all models generate diverse errors, their alignment to human submissions diverges: Claude Sonnet 4 achieves the most balanced performance.
  arXiv  Detail & Related papers  (2026-06-12T04:51:49Z)
• Automated Grading of Handwritten Mathematics Using Vision-Capable LLMs [2.727845763576068]
  Vision-capable large language models (LLMs) offer new opportunities.<n>We present an empirical evaluation of an LLM-based grader for handwritten mathematical work using instructor-defined rubrics.
  arXiv  Detail & Related papers  (2026-05-18T19:06:15Z)
• A Multi-Agent Approach to Validate and Refine LLM-Generated Personalized Math Problems [5.687145473906]
  We propose a framework that formalizes personalization as an iterative generate--validate--revise process.<n>We use four specialized validator agents targeting the criteria of solvability, realism, readability, and authenticity.<n>We evaluate our framework on 600 problems drawn from a popular online mathematics homework platform, ASSISTments.
  arXiv  Detail & Related papers  (2026-04-06T20:47:01Z)
• Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models [86.45058529521258]
  OlymMATH is a novel Olympiad-level mathematical benchmark designed to rigorously test the complex reasoning capabilities of LLMs.<n>OlymMATH features 200 meticulously curated problems, each manually verified and available in parallel English and Chinese versions.
  arXiv  Detail & Related papers  (2025-03-27T11:20:17Z)
• MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection [53.325457460187046]
  We introduce MathAgent, a novel Mixture-of-Math-Agent framework designed specifically to address these challenges.<n>MathAgent decomposes error detection into three phases, each handled by a specialized agent.<n>We evaluate MathAgent on real-world educational data, demonstrating approximately 5% higher accuracy in error step identification.
  arXiv  Detail & Related papers  (2025-03-23T16:25:08Z)
• Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Framework [79.40678802098026]
  Math Word Problems serve as a crucial benchmark for evaluating Large Language Models' reasoning abilities.<n>Current error classification methods rely on static and predefined categories.<n>We propose Error-Aware Prompting (EAP) that incorporates common error patterns as explicit guidance.
  arXiv  Detail & Related papers  (2025-01-26T16:17:57Z)
• ProcessBench: Identifying Process Errors in Mathematical Reasoning [62.80402845414901]
  We introduce ProcessBench for measuring the ability to identify erroneous steps in mathematical reasoning.<n>ProcessBench consists of 3,400 test cases, primarily focused on competition- and Olympiad-level math problems.<n>We conduct extensive evaluation on ProcessBench, involving two types of models: process reward models (PRMs) and critic models.
  arXiv  Detail & Related papers  (2024-12-09T15:11:40Z)
• ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection [60.297079601066784]
  We introduce ErrorRadar, the first benchmark designed to assess MLLMs' capabilities in error detection. ErrorRadar evaluates two sub-tasks: error step identification and error categorization. It consists of 2,500 high-quality multimodal K-12 mathematical problems, collected from real-world student interactions. Results indicate significant challenges still remain, as GPT-4o with best performance is still around 10% behind human evaluation.
  arXiv  Detail & Related papers  (2024-10-06T14:59:09Z)
• DiVERT: Distractor Generation with Variational Errors Represented as Text for Math Multiple-choice Questions [42.148511874019256]
  We introduce DiVERT, a novel variational approach that learns an interpretable representation of errors behind distractors in math multiple-choice questions (MCQs) We show that DiVERT, despite using a base open-source LLM with 7B parameters, outperforms state-of-the-art approaches using GPT-4o on downstream distractor generation. We also conduct a human evaluation with math educators and find that DiVERT leads to error labels that are of comparable quality to human-authored ones.
  arXiv  Detail & Related papers  (2024-06-27T17:37:31Z)
• Can Large Language Models Replicate ITS Feedback on Open-Ended Math Questions? [3.7399138244928145]
  We study the capabilities of large language models to generate feedback for open-ended math questions. We find that open-source and proprietary models both show promise in replicating the feedback they see during training, but do not generalize well to previously unseen student errors.
  arXiv  Detail & Related papers  (2024-05-10T11:53:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.