Estimating Item Difficulty with Large Language Models as Experts

• URL: http://arxiv.org/abs/2605.18562v1
• Date: Mon, 18 May 2026 15:42:13 GMT
• Title: Estimating Item Difficulty with Large Language Models as Experts
• Abstract summary: This study evaluated three off-the-shelf language models as difficulty raters for newly created items without access to response data.<n>The study used a full factorial design crossing three factors: judgement format (absolute vs pairwise), decision type (hard decisions vs token-probability-based estimates), and prompting strategy.<n>Across domains, LLM-based estimates exhibited moderate to strong positive correlations with empirical item difficulties.
• Score: 0.44101646956991475
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Accurate estimates of item difficulty are essential for valid assessment and effective adaptive learning. However, for newly created tasks, response data are typically unavailable. Pretesting and expert judgement can be costly and slow, while machine learning methods often require large labelled training datasets. Recent work suggests that large language models (LLMs) may help. However, there is limited evidence on the elicitation procedures and prompt configurations used to emulate experts for difficulty estimation. This study addresses this gap by evaluating three off-the-shelf LLMs as difficulty raters for newly created items without access to response data. Using an item bank from an online learning system, the study examined 6 domains of primary-school mathematics, with empirical difficulty estimates treated as empirical reference. The study used a full factorial design crossing three factors: judgement format (absolute vs pairwise), decision type (hard decisions vs token-probability-based estimates), and prompting strategy (zero-shot vs few-shot). LLM-derived difficulty estimates were compared with empirical difficulties using Spearman rank correlations. Across domains, LLM-based estimates exhibited moderate to strong positive correlations with empirical item difficulties. For simpler arithmetic tasks, some configurations approached the upper end of the accuracy range reported for human experts in previous research. Pairwise comparison consistently outperformed absolute judgement in the absence of additional refinements. However, when token-level probabilities were incorporated and examples of items with known empirical difficulty were provided, the absolute judgement configuration likewise demonstrated moderate-to-high alignment. The study positions LLMs as a promising tool for initial item calibration and offers insights into effective workflow configuration.

Related papers

• The Easy Trap: Why LLMs Underestimate Misconception-Driven Difficulty [4.205487728047868]
  Large language models (LLMs) are increasingly used for estimating item difficulty in educational assessment.<n>This study investigates the alignment between LLM-generated difficulty ratings and empirical student performance on basic mathematics tasks.
  arXiv  Detail & Related papers  (2026-06-22T09:53:25Z)
• RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty [102.02839046225468]
  RankLLM is a novel framework designed to quantify both question difficulty and model competency.<n>We evaluate 30 models on 35,550 questions across multiple domains.
  arXiv  Detail & Related papers  (2026-02-12T21:28:46Z)
• Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations [36.23612429926861]
  We investigate the predictive value of open-source large language models (LLMs) for evaluating the difficulty of math questions for real-world students.<n>We simulate a "classroom" of 4th, 8th, or 12th grade students by prompting the LLM to role-play students of varying proficiency levels.<n>We observe correlations as high as 0.75, 0.76, and 0.82 for grades 4, 8, and 12, respectively.
  arXiv  Detail & Related papers  (2026-01-15T00:25:01Z)
• Estimating problem difficulty without ground truth using Large Language Model comparisons [4.599673637363014]
  We propose a new method for estimating problem difficulty, LLM compare.<n>An LLM performs pairwise difficulty comparisons, and then Bradley-Terry scores are computed based on the outcomes.<n>Our work represents a significant step towards replacing time-consuming human annotations and synthetic data generation.
  arXiv  Detail & Related papers  (2025-12-16T09:13:56Z)
• Revisiting Generalization Across Difficulty Levels: It's Not So Easy [11.203451380580868]
  We investigate how well large language models generalize across different task difficulties.<n>We show that training on either easy or hard data cannot achieve consistent improvements across the full range of difficulties.
  arXiv  Detail & Related papers  (2025-11-26T18:59:57Z)
• The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations [33.65540900920885]
  Estimating the difficulty of input questions as perceived by large language models (LLMs) is essential for accurate performance evaluation and adaptive inference.<n>We propose a novel approach for difficulty estimation that leverages only the hidden representations produced by the target LLM.
  arXiv  Detail & Related papers  (2025-09-16T09:38:41Z)
• Curse of Knowledge: When Complex Evaluation Context Benefits yet Biases LLM Judges [72.3356133063925]
  The paradigm of large language models (LLMs) as judges has emerged as a scalable solution, yet prior work primarily focuses on simple settings.<n>Our in-depth analysis offers crucial insights for improving the accuracy and verifiability of evaluation signals.
  arXiv  Detail & Related papers  (2025-09-03T15:48:33Z)
• Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding [53.63482987410292]
  We present a self-adaptive curriculum learning paradigm that prioritizes fine-tuning examples based on difficulty scores predicted by pre-trained language models.<n>We evaluate our method on four natural language understanding (NLU) datasets covering both binary and multi-class classification tasks.
  arXiv  Detail & Related papers  (2025-07-13T19:36:17Z)
• Truth in the Few: High-Value Data Selection for Efficient Multi-Modal Reasoning [71.3533541927459]
  We propose a novel data selection paradigm termed Activation Reasoning Potential (RAP)<n>RAP identifies cognitive samples by estimating each sample's potential to stimulate genuine multi-modal reasoning.<n>Our RAP method consistently achieves superior performance using only 9.3% of the training data, while reducing computational costs by over 43%.
  arXiv  Detail & Related papers  (2025-06-05T08:40:24Z)
• Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms [0.0]
  Estimating item difficulty through field-testing is often resource-intensive and time-consuming.<n>The present research examines the feasibility of using an Large Language Models (LLMs) to predict item difficulty for K-5 mathematics and reading assessment items.
  arXiv  Detail & Related papers  (2025-04-09T00:04:07Z)
• Reliable and Efficient Amortized Model-based Evaluation [57.6469531082784]
  The average score across a wide range of benchmarks provides a signal that helps guide the use of language models in practice.<n>A popular attempt to lower the cost is to compute the average score on a subset of the benchmark.<n>This approach often renders an unreliable measure of LM performance because the average score is often confounded with the difficulty of the questions in the benchmark subset.<n>We train a model that predicts question difficulty from its content, enabling a reliable measurement at a fraction of the cost.
  arXiv  Detail & Related papers  (2025-03-17T16:15:02Z)
• Subtle Errors in Reasoning: Preference Learning via Error-injected Self-editing [59.405145971637204]
  We propose a novel preference learning framework called eRror-Injected Self-Editing (RISE)<n>RISE injects predefined subtle errors into pivotal tokens in reasoning or steps to construct hard pairs for error mitigation.<n>Experiments validate the effectiveness of RISE, with preference learning on Qwen2-7B-Instruct yielding notable improvements of 3.0% on GSM8K and 7.9% on MATH with only 4.5K training samples.
  arXiv  Detail & Related papers  (2024-10-09T07:43:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.