Catching The Correct Answer Trap: Characterising AI Tutor Blind Spots When Analysing Student Reasoning

• URL: http://arxiv.org/abs/2605.23925v1
• Date: Mon, 20 Apr 2026 10:53:48 GMT
• Title: Catching The Correct Answer Trap: Characterising AI Tutor Blind Spots When Analysing Student Reasoning
• Abstract summary: We study a failure mode we call the correct answer trap (CAT): models under-detect misconceptions when students reach a correct answer via flawed reasoning.<n>Analysing real student responses from the Eedi mathematics platform, we show that 71% of these failures concentrate in just two question types.
• Score: 2.6352293843915606
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Intelligent tutoring systems increasingly provide automated feedback on student work, but robust feedback requires assessing reasoning, not only final answers. We study a failure mode we call the correct answer trap (CAT): models under-detect misconceptions when students reach a correct answer via flawed reasoning. Analysing real student responses from the Eedi mathematics platform, we show that 71% of these failures concentrate in just two question types, both sharing a common structure where flawed reasoning happens to produce the correct numerical answer. Comparing a fine-tuned T5 with a frontier large language model, we find that improved capabilities reduce but do not eliminate the problem (84% vs 57% detection accuracy). Even the best-performing model generates roughly four false alarms for every genuine detection, making stand-alone screening impractical at realistic class sizes. Our findings demonstrate that high overall accuracy can mask critical failures in reasoning assessment, and that careful analysis of student reasoning still benefits from human judgment.

Related papers

• The Correct Answer Trap: Pedagogically-Grounded Detection and Feedback for Hidden Misconceptions [2.6352293843915606]
  We investigate automatic detection of hidden misconceptions using 20,964 real student responses from the Eedi mathematics platform.<n>We present a graduated assessment that separates answer correctness from method validity, and propose a detect-verify-escalate pipeline.<n>Two deployment modes adapt the pipeline: a teacher dashboard where the system filters a review queue, and an autonomous tutor where flags trigger low-cost formative follow-up.
  arXiv  Detail & Related papers  (2026-06-22T11:51:17Z)
• Pause or Fabricate? Training Language Models for Grounded Reasoning [50.104657152302956]
  We propose Grounded Reasoning via Interactive Reinforcement Learning (GRIL) for grounded reasoning under incomplete information.<n>GRIL decomposes the reasoning process into two stages: clarify and pause, which identifies whether the available information is sufficient, and grounded reasoning, which performs task solving once the necessary premises are established.<n>Experiments on GSM8K-Insufficient and MetaMATH-Insufficient show that GRIL significantly improves premise detection (up to 45%), leading to a 30% increase in task success while reducing average response length by over 20%.
  arXiv  Detail & Related papers  (2026-04-21T16:45:29Z)
• Can LLMs Model Incorrect Student Reasoning? A Case Study on Distractor Generation [46.210941814500906]
  We examine how large language models (LLMs) reason about misconceptions when generating multiple-choice distractors.<n>Our structured analysis reveals a surprising alignment between their processes and best practices.
  arXiv  Detail & Related papers  (2026-03-16T17:09:41Z)
• Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course [0.7515394929245942]
  This paper is a student-led field experiment embedded as a midterm project in UNIV 182 (AI4All) at George Mason University.<n>Students designed their own reasoning tasks, ran them on widely used consumer chatbots representative of current capabilities, and evaluated both (i) answer correctness and (ii) the validity of the reasoning.<n>OpenAI GPT-5 and Claude 4.5 achieved the highest mean answer accuracy (86.2% and 83.8%), followed by Grok 4 (82.5%) and Perplexity (73.1%)
  arXiv  Detail & Related papers  (2025-12-28T22:51:25Z)
• Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors [58.65143578052761]
  This paper presents a new method, MISTAKE, that constructs high-quality synthetic examples of reasoning errors.<n>We evaluate MISTAKE on three educational tasks and find that it results in (1) higher accuracy when simulating incorrect student answers.
  arXiv  Detail & Related papers  (2025-10-13T15:10:38Z)
• Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Think [51.0691253204425]
  We analyze intermediate reasoning steps, termed subthoughts, to answer two questions: Does the final answer reliably represent the model's optimal conclusion?<n>Our approach involves segmenting a reasoning trace into sequential subthoughts based on linguistic cues.<n>We find that aggregating these answers by selecting the most frequent one (the mode) often yields significantly higher accuracy compared to relying solely on the answer derived from the original complete trace.
  arXiv  Detail & Related papers  (2025-04-29T12:39:07Z)
• FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving [90.88021670297664]
  FINEREASON is a logic-puzzle benchmark for evaluation of large language models' reasoning capabilities.<n>We introduce two tasks: state checking, and state transition, for a comprehensive evaluation of how models assess the current situation and plan the next move.<n>We show that models trained on our state checking and transition data demonstrate gains in math reasoning by up to 5.1% on GSM8K.
  arXiv  Detail & Related papers  (2025-02-27T16:23:25Z)
• Large Language Models and Mathematical Reasoning Failures [1.6114012813668932]
  This paper investigates the mathematical reasoning capabilities of large language models (LLMs) using 50 newly constructed high-school-level word problems.<n>We rigorously analyze both final answers and solution steps to identify reasoning failures.<n>We find that while newer models (e.g., o3-mini, deepseek-r1) achieve higher accuracy, all models exhibit errors in spatial reasoning, strategic planning, and arithmetic.
  arXiv  Detail & Related papers  (2025-02-17T09:07:32Z)
• Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors [78.53699244846285]
  Large language models (LLMs) present an opportunity to scale high-quality personalized education to all. LLMs struggle to precisely detect student's errors and tailor their feedback to these errors. Inspired by real-world teaching practice where teachers identify student errors and customize their response based on them, we focus on verifying student solutions.
  arXiv  Detail & Related papers  (2024-07-12T10:11:40Z)
• Self-Contradictory Reasoning Evaluation and Detection [31.452161594896978]
  We investigate self-contradictory (Self-Contra) reasoning, where the model reasoning does not support its answers. We find that LLMs often contradict themselves in reasoning tasks involving contextual information understanding or commonsense. We find that GPT-4 can detect Self-Contra with a 52.2% F1 score, much lower compared to 66.7% for humans.
  arXiv  Detail & Related papers  (2023-11-16T06:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.