Can LLMs Model Incorrect Student Reasoning? A Case Study on Distractor Generation

• URL: http://arxiv.org/abs/2603.15547v1
• Date: Mon, 16 Mar 2026 17:09:41 GMT
• Title: Can LLMs Model Incorrect Student Reasoning? A Case Study on Distractor Generation
• Abstract summary: We examine how large language models (LLMs) reason about misconceptions when generating multiple-choice distractors.<n>Our structured analysis reveals a surprising alignment between their processes and best practices.
• Score: 46.210941814500906
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modeling plausible student misconceptions is critical for AI in education. In this work, we examine how large language models (LLMs) reason about misconceptions when generating multiple-choice distractors, a task that requires modeling incorrect yet plausible answers by coordinating solution knowledge, simulating student misconceptions, and evaluating plausibility. We introduce a taxonomy for analyzing the strategies used by state-of-the-art LLMs, examining their reasoning procedures and comparing them to established best practices in the learning sciences. Our structured analysis reveals a surprising alignment between their processes and best practices: the models typically solve the problem correctly first, then articulate and simulate multiple potential misconceptions, and finally select a set of distractors. An analysis of failure modes reveals that errors arise primarily from failures in recovering the correct solution and selecting among response candidates, rather than simulating errors or structuring the process. Consistent with these results, we find that providing the correct solution in the prompt improves alignment with human-authored distractors by 8%, highlighting the critical role of anchoring to the correct solution when generating plausible incorrect student reasoning. Overall, our analysis offers a structured and interpretable lens into LLMs' ability to model incorrect student reasoning and produce high-quality distractors.

Related papers

• Misconception Acquisition Dynamics in Large Language Models [41.92200721590384]
  We study two misconception-aware models: the Novice Student Misconception Model and the Expert Tutor Misconception Model.<n>We develop MalAlgoLib, a library that generates algebra problems with correct solution traces and misconception-specific erroneous traces.<n>Our experiments reveal that the student and the tutor model exhibit fundamentally different misconception acquisition dynamics.
  arXiv  Detail & Related papers  (2026-04-01T12:25:20Z)
• Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors [58.65143578052761]
  This paper presents a new method, MISTAKE, that constructs high-quality synthetic examples of reasoning errors.<n>We evaluate MISTAKE on three educational tasks and find that it results in (1) higher accuracy when simulating incorrect student answers.
  arXiv  Detail & Related papers  (2025-10-13T15:10:38Z)
• Improving Rationality in the Reasoning Process of Language Models through Self-playing Game [25.193698725021108]
  We design a Critic-Discernment Game (CDG) in which a prover first provides a solution to a given problem and is subsequently challenged by critiques of its solution.<n>The objective of the prover is to maintain the correct answer when faced with misleading comments, while correcting errors in response to constructive feedback.<n>Our experiments on tasks involving mathematical reasoning, stepwise error detection, self-correction, and long-chain reasoning demonstrate that CDG training can significantly improve the ability of well-aligned LLMs to comprehend their reasoning process.
  arXiv  Detail & Related papers  (2025-06-28T15:11:23Z)
• Large Language Models and Mathematical Reasoning Failures [1.6114012813668932]
  This paper investigates the mathematical reasoning capabilities of large language models (LLMs) using 50 newly constructed high-school-level word problems.<n>We rigorously analyze both final answers and solution steps to identify reasoning failures.<n>We find that while newer models (e.g., o3-mini, deepseek-r1) achieve higher accuracy, all models exhibit errors in spatial reasoning, strategic planning, and arithmetic.
  arXiv  Detail & Related papers  (2025-02-17T09:07:32Z)
• Self-supervised Analogical Learning using Language Models [59.64260218737556]
  We propose SAL, a self-supervised analogical learning framework.<n> SAL mimics the human analogy process and trains models to explicitly transfer high-quality symbolic solutions.<n>We show that the resulting models outperform base language models on a wide range of reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-02-03T02:31:26Z)
• LLM-based Cognitive Models of Students with Misconceptions [55.29525439159345]
  This paper investigates whether Large Language Models (LLMs) can be instruction-tuned to meet this dual requirement. We introduce MalAlgoPy, a novel Python library that generates datasets reflecting authentic student solution patterns. Our insights enhance our understanding of AI-based student models and pave the way for effective adaptive learning systems.
  arXiv  Detail & Related papers  (2024-10-16T06:51:09Z)
• Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors [78.53699244846285]
  Large language models (LLMs) present an opportunity to scale high-quality personalized education to all. LLMs struggle to precisely detect student's errors and tailor their feedback to these errors. Inspired by real-world teaching practice where teachers identify student errors and customize their response based on them, we focus on verifying student solutions.
  arXiv  Detail & Related papers  (2024-07-12T10:11:40Z)
• A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning [73.77088902676306]
  We take a closer look at the self-verification abilities of large language models (LLMs) in the context of logical reasoning. Our main findings suggest that existing LLMs could struggle to identify fallacious reasoning steps accurately and may fall short of guaranteeing the validity of self-verification methods.
  arXiv  Detail & Related papers  (2023-11-14T07:13:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.