Error as a Lens: Probing LLM Reasoning through Synthetic Misconception Generation

• URL: http://arxiv.org/abs/2605.29007v1
• Date: Wed, 27 May 2026 19:04:10 GMT
• Title: Error as a Lens: Probing LLM Reasoning through Synthetic Misconception Generation
• Abstract summary: We present a framework that generates errors targeted to a five-class taxonomy adapted from the revised Bloom's taxonomy.<n>The framework yields a reusable recipe for building class-stratified synthetic error datasets.
• Score: 3.6825041714447457
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Personalized tutoring, teacher training, and education research need access to \emph{targeted} synthetic misconceptions, but privacy and IRB constraints make labelled corpora of real student errors scarce. LLMs could in principle generate synthetic errors at scale, but producing an arbitrary wrong answer is easy for a modern LLM while producing one that matches a specified cognitive failure mode is much harder. We present a framework that generates errors targeted to a five-class taxonomy adapted from the revised Bloom's taxonomy, evaluated on questions from the TheoremQA dataset. A Generation Agent (GA) drafts a candidate erroneous solution conditioned on a target class, and an Examination Agent (EA) judges whether the draft is incorrect and class-consistent. The framework yields a reusable recipe for building class-stratified synthetic error datasets where authentic student corpora are unavailable. As a secondary diagnostic, targeted error generation is substantially harder than free-form incorrect-answer generation, and answer-grounding contributes more than expanded examples or external textbook content.

Related papers

• Automated Grading of Handwritten Mathematics Using Vision-Capable LLMs [2.727845763576068]
  Vision-capable large language models (LLMs) offer new opportunities.<n>We present an empirical evaluation of an LLM-based grader for handwritten mathematical work using instructor-defined rubrics.
  arXiv  Detail & Related papers  (2026-05-18T19:06:15Z)
• How LLMs Fail and Generalize in RTL Coding for Hardware Design? [56.361436215029045]
  We introduce a new error taxonomy grounded in problem solvability, inspired by cognitive theory.<n>Our taxonomy categorizes failures into syntactic, semantic, solvable functional, and unsolvable functional types.
  arXiv  Detail & Related papers  (2026-04-26T14:34:49Z)
• From Subsumption to Satisfiability: LLM-Assisted Active Learning for OWL Ontologies [1.1351175212666613]
  In active learning, membership queries (MQs) allow a learner to pose questions to a teacher.<n>We reformulate each candidate axiom into its corresponding counter-concept and verbalise it in controlled natural language.<n>We introduce Large Language Models (LLMs) as a third component that provides real-world examples approximating an instance of the counter-concept.
  arXiv  Detail & Related papers  (2026-04-17T20:05:30Z)
• Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math [55.83696908107408]
  We introduce ScratchMath, a novel benchmark for explaining and classifying errors in authentic handwritten mathematics scratchwork.<n>Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students.<n>We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts.
  arXiv  Detail & Related papers  (2026-03-26T02:57:20Z)
• LEMMA: Learning from Errors for MatheMatical Advancement in LLMs [33.571479131705075]
  We introduce Learning from Errors for Mathematical Advancement (LEMMA) to enhance large language models' reasoning ability.<n> LEMMA constructs data consisting of an incorrect solution with an erroneous step and a reflection connection to a correct solution for fine-tuning.<n> Experimental results demonstrate that LEMMA achieves significant performance improvements over other strong baselines.
  arXiv  Detail & Related papers  (2025-03-21T17:59:10Z)
• Teaching Your Models to Understand Code via Focal Preference Alignment [70.71693365502212]
  In existing approaches, a set of n candidate solutions is evaluated based on test case success rates.<n>Because this approach aligns entire failing code blocks rather than pinpointing specific errors, it lacks the granularity necessary to capture meaningful error-correction relationships.<n>We propose Target-DPO, a new preference alignment framework that mimics human iterative debug to refine Code LLMs.
  arXiv  Detail & Related papers  (2025-03-04T16:56:34Z)
• Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Framework [79.40678802098026]
  Math Word Problems serve as a crucial benchmark for evaluating Large Language Models' reasoning abilities.<n>Current error classification methods rely on static and predefined categories.<n>We propose Error-Aware Prompting (EAP) that incorporates common error patterns as explicit guidance.
  arXiv  Detail & Related papers  (2025-01-26T16:17:57Z)
• Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Students [4.949067768845775]
  Large language models (LLMs) present an exciting opportunity for generating synthetic classroom data. This research paper examines the distribution of bugs generated by LLMs in contrast to those produced by computing students.
  arXiv  Detail & Related papers  (2024-10-11T18:51:58Z)
• Subtle Errors in Reasoning: Preference Learning via Error-injected Self-editing [59.405145971637204]
  We propose a novel preference learning framework called eRror-Injected Self-Editing (RISE)<n>RISE injects predefined subtle errors into pivotal tokens in reasoning or steps to construct hard pairs for error mitigation.<n>Experiments validate the effectiveness of RISE, with preference learning on Qwen2-7B-Instruct yielding notable improvements of 3.0% on GSM8K and 7.9% on MATH with only 4.5K training samples.
  arXiv  Detail & Related papers  (2024-10-09T07:43:38Z)
• Understanding and Mitigating Classification Errors Through Interpretable Token Patterns [58.91023283103762]
  Characterizing errors in easily interpretable terms gives insight into whether a classifier is prone to making systematic errors. We propose to discover those patterns of tokens that distinguish correct and erroneous predictions. We show that our method, Premise, performs well in practice.
  arXiv  Detail & Related papers  (2023-11-18T00:24:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.