Finding Common Mistakes In Modelling With Mathematical Formalisms Using LLMs

• URL: http://arxiv.org/abs/2609.17111v1
• Date: Tue, 15 Sep 2026 12:40:27 GMT
• Title: Finding Common Mistakes In Modelling With Mathematical Formalisms Using LLMs
• Abstract summary: We present a tool-supported workflow that allows to identify candidates for common mistakes that explain many student mistakes in large educational data sets.<n>The visualization is designed to help researchers to identify common modelling mistakes.<n>We show that this approach works well by reproducing common mistakes in propositional logic modelling that were identified by hand in the literature.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modelling with mathematical formalisms like logical formulas, mathematical equations, or regular expressions is an important yet challenging task for students of computer science and other STEM disciplines. Identifying common mistakes occurring in this context is an important step towards helping struggling students by providing targeted high-quality feedback, e.g. in interactive learning systems. We present a tool-supported workflow that allows to (1) identify candidates for common mistakes that explain many student mistakes in large educational data sets, (2) cluster candidates according to similarities, and (3) visualize resulting clusters for instructors and CS education researchers. The visualization is designed to help researchers to identify common modelling mistakes. The candidates for common mistakes are represented by bug fixing transformations that translate incorrect formalizations into correct formalizations; they are generated by an LLM and validated algorithmically. We show that this approach works well by reproducing common mistakes in propositional logic modelling that were identified by hand in the literature; showing that, unlike other algorithmic approaches, the LLM-based approach is suitable for very large sets of data; and applying it to multiple other formalisms to showcase it generalizes beyond propositional logic.

Related papers

• Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math [55.83696908107408]
  We introduce ScratchMath, a novel benchmark for explaining and classifying errors in authentic handwritten mathematics scratchwork.<n>Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students.<n>We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts.
  arXiv  Detail & Related papers  (2026-03-26T02:57:20Z)
• Automatic Syntax Error Repair for Discrete Controller Synthesis using Large Language Model [8.741815980649667]
  This paper presents an automated approach that leverages Large Language Models (LLMs) to repair syntax errors in DCS models.<n>It equips the LLM with DCS-specific domain knowledge, including formal grammar rules and illustrative examples, to guide accurate corrections.<n>It achieves a speedup of 3.46 times compared to human developers.
  arXiv  Detail & Related papers  (2025-12-08T07:57:15Z)
• Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors [58.65143578052761]
  This paper presents a new method, MISTAKE, that constructs high-quality synthetic examples of reasoning errors.<n>We evaluate MISTAKE on three educational tasks and find that it results in (1) higher accuracy when simulating incorrect student answers.
  arXiv  Detail & Related papers  (2025-10-13T15:10:38Z)
• Step-Wise Formal Verification for LLM-Based Mathematical Problem Solving [3.2233767737586674]
  Large Language Models (LLMs) have demonstrated formidable capabilities in solving mathematical problems.<n>This paper proposes a framework, MATH-VF, which includes a Formalizer and a Critic.<n>We evaluate our framework on widely used mathematical benchmarks: MATH500 and ProcessBench.
  arXiv  Detail & Related papers  (2025-05-27T08:21:07Z)
• Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Framework [79.40678802098026]
  Math Word Problems serve as a crucial benchmark for evaluating Large Language Models' reasoning abilities.<n>Current error classification methods rely on static and predefined categories.<n>We propose Error-Aware Prompting (EAP) that incorporates common error patterns as explicit guidance.
  arXiv  Detail & Related papers  (2025-01-26T16:17:57Z)
• LLM-based Cognitive Models of Students with Misconceptions [55.29525439159345]
  This paper investigates whether Large Language Models (LLMs) can be instruction-tuned to meet this dual requirement. We introduce MalAlgoPy, a novel Python library that generates datasets reflecting authentic student solution patterns. Our insights enhance our understanding of AI-based student models and pave the way for effective adaptive learning systems.
  arXiv  Detail & Related papers  (2024-10-16T06:51:09Z)
• LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback [71.95402654982095]
  We propose Math-Minos, a natural language feedback-enhanced verifier. Our experiments reveal that a small set of natural language feedback can significantly boost the performance of the verifier.
  arXiv  Detail & Related papers  (2024-06-20T06:42:27Z)
• Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models [5.162225137921625]
  Large language models (LLMs) have recently demonstrated surprising performance for a range of computing tasks. We investigate the performance of two popular LLMs, GPT-3 and GPT-4, for detecting and providing a novice-friendly explanation of logic errors.
  arXiv  Detail & Related papers  (2023-11-27T17:28:33Z)
• Understanding and Mitigating Classification Errors Through Interpretable Token Patterns [58.91023283103762]
  Characterizing errors in easily interpretable terms gives insight into whether a classifier is prone to making systematic errors. We propose to discover those patterns of tokens that distinguish correct and erroneous predictions. We show that our method, Premise, performs well in practice.
  arXiv  Detail & Related papers  (2023-11-18T00:24:26Z)
• Label-Descriptive Patterns and their Application to Characterizing Classification Errors [31.272875287136426]
  State-of-the-art deep learning methods achieve human-like performance on many tasks, but make errors nevertheless. Characterizing these errors in easily interpretable terms gives insight into whether a model is prone to making systematic errors, but also gives a way to act and improve the model. In this paper we propose a method that allows us to do so for arbitrary classifiers by mining a small set of patterns that together succinctly describe the input data that is partitioned according to correctness of prediction.
  arXiv  Detail & Related papers  (2021-10-18T19:42:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.