The Easy Trap: Why LLMs Underestimate Misconception-Driven Difficulty

• URL: http://arxiv.org/abs/2607.26067v1
• Date: Mon, 22 Jun 2026 09:53:25 GMT
• Title: The Easy Trap: Why LLMs Underestimate Misconception-Driven Difficulty
• Abstract summary: Large language models (LLMs) are increasingly used for estimating item difficulty in educational assessment.<n>This study investigates the alignment between LLM-generated difficulty ratings and empirical student performance on basic mathematics tasks.
• Score: 4.205487728047868
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) are increasingly used for estimating item difficulty in educational assessment. However, it remains unclear whether such estimates reflect how learners actually experience difficulty. This study investigates the alignment between LLM-generated difficulty ratings and empirical student performance on basic mathematics tasks. Four widely used LLM-based systems generated difficulty ratings on a 1-100 scale for 32 arithmetic items across multiple runs (N = 640 ratings). These were compared with empirical difficulty derived from responses of 770 Indonesian undergraduates using Classical Test Theory (CTT) and Item Response Theory (2PL). Results show moderate rank correlations (Spearman's rho = 0.52-0.70), indicating that LLMs capture coarse ordering of item difficulty. However, substantial and systematic misalignment emerges in fraction items. Several items consistently rated as easy by LLMs were among the most difficult for students, such as an item with only 34.16% correct for 100 : 1/2. We argue that LLMs approximate curricular difficulty, or what should be easy based on instructional sequencing, rather than cognitive difficulty driven by learner misconceptions. This leads to systematic underestimation of misconception-driven items, a phenomenon we term the Easy Trap. These findings highlight a critical limitation of LLM-based difficulty estimation and suggest that relying on such estimates without empirical grounding may introduce bias in assessment design and adaptive systems.

Related papers

• Estimating Item Difficulty with Large Language Models as Experts [0.44101646956991475]
  This study evaluated three off-the-shelf language models as difficulty raters for newly created items without access to response data.<n>The study used a full factorial design crossing three factors: judgement format (absolute vs pairwise), decision type (hard decisions vs token-probability-based estimates), and prompting strategy.<n>Across domains, LLM-based estimates exhibited moderate to strong positive correlations with empirical item difficulties.
  arXiv  Detail & Related papers  (2026-05-18T15:42:13Z)
• Rethinking Easy-to-Hard: Limits of Curriculum Learning in Post-Training for Deductive Reasoning [61.81622271655058]
  We present a systematic empirical study of curriculum-based post-training of large language models.<n>We find no robust advantage in difficulty-based sequencing over standard random sampling in either accuracy or response length.<n>Our study suggests that, in the context of deductive reasoning, the specific ordering of training examples plays a negligible role in achieving compositional generalization.
  arXiv  Detail & Related papers  (2026-03-28T10:43:36Z)
• RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty [102.02839046225468]
  RankLLM is a novel framework designed to quantify both question difficulty and model competency.<n>We evaluate 30 models on 35,550 questions across multiple domains.
  arXiv  Detail & Related papers  (2026-02-12T21:28:46Z)
• Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations [36.23612429926861]
  We investigate the predictive value of open-source large language models (LLMs) for evaluating the difficulty of math questions for real-world students.<n>We simulate a "classroom" of 4th, 8th, or 12th grade students by prompting the LLM to role-play students of varying proficiency levels.<n>We observe correlations as high as 0.75, 0.76, and 0.82 for grades 4, 8, and 12, respectively.
  arXiv  Detail & Related papers  (2026-01-15T00:25:01Z)
• Revisiting Generalization Across Difficulty Levels: It's Not So Easy [11.203451380580868]
  We investigate how well large language models generalize across different task difficulties.<n>We show that training on either easy or hard data cannot achieve consistent improvements across the full range of difficulties.
  arXiv  Detail & Related papers  (2025-11-26T18:59:57Z)
• Probing the Difficulty Perception Mechanism of Large Language Models [31.945071671041465]
  We investigate whether large language models implicitly encode problem difficulty in their internal representations.<n>We locate the specific attention heads of the final Transformer layer.<n>Experiments provide practical support for using LLMs as automatic difficulty annotators.
  arXiv  Detail & Related papers  (2025-10-07T14:24:32Z)
• Measuring Scalar Constructs in Social Science with LLMs [48.92998035333579]
  We evaluate approaches to measuring scalar constructs in large language models.<n>We find that pairwise comparisons produce better measurements than simply prompting the LLM to directly output the scores.<n>Finetuning smaller models with as few as 1,000 training pairs can match or exceed the performance of prompted LLMs.
  arXiv  Detail & Related papers  (2025-09-03T08:19:13Z)
• Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms [0.0]
  Estimating item difficulty through field-testing is often resource-intensive and time-consuming.<n>The present research examines the feasibility of using an Large Language Models (LLMs) to predict item difficulty for K-5 mathematics and reading assessment items.
  arXiv  Detail & Related papers  (2025-04-09T00:04:07Z)
• Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling [69.57918638435491]
  Test-Time Scaling is an important method for improving the performance of Large Language Models.<n>This paper focuses on two core questions: What is the optimal approach to scale test-time computation across different policy models, PRMs, and problem difficulty levels?<n>We show that with our compute-optimal TTS strategy, extremely small policy models can outperform larger models.
  arXiv  Detail & Related papers  (2025-02-10T17:30:23Z)
• Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems [50.76385564061713]
  Chain-of-Thought (CoT) prompting has enhanced the performance of Large Language Models (LLMs) across various reasoning tasks.<n>CoT usually suffers from three pitfalls: semantic misunderstanding errors, calculation errors, and step-missing errors.<n>We propose Deeply Understanding the Problems (DUP) to improve the LLMs' math problem-solving ability by addressing semantic misunderstanding errors.
  arXiv  Detail & Related papers  (2024-04-23T12:16:05Z)
• GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers [68.77382332826167]
  Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks. One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly. This motivates us to evaluate the robustness of LLMs' math reasoning capability by testing a wide range of question variations.
  arXiv  Detail & Related papers  (2024-02-29T15:26:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.