Synthetic Student Responses: LLM-Extracted Features for IRT Difficulty Parameter Estimation

• URL: http://arxiv.org/abs/2602.00034v1
• Date: Sun, 18 Jan 2026 18:19:13 GMT
• Title: Synthetic Student Responses: LLM-Extracted Features for IRT Difficulty Parameter Estimation
• Authors: Matias Hoyl,
• Abstract summary: We investigate whether Item Response Theory difficulty parameters can be accurately estimated without student testing.<n>Our approach combines traditional linguistic features with pedagogical insights extracted using Large Language Models (LLMs)<n>Using a dataset of over 250,000 student responses to mathematics questions, our model achieves a Pearson correlation of approximately 0.78 between predicted and actual difficulty parameters on completely unseen questions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Educational assessment relies heavily on knowing question difficulty, traditionally determined through resource-intensive pre-testing with students. This creates significant barriers for both classroom teachers and assessment developers. We investigate whether Item Response Theory (IRT) difficulty parameters can be accurately estimated without student testing by modeling the response process and explore the relative contribution of different feature types to prediction accuracy. Our approach combines traditional linguistic features with pedagogical insights extracted using Large Language Models (LLMs), including solution step count, cognitive complexity, and potential misconceptions. We implement a two-stage process: first training a neural network to predict how students would respond to questions, then deriving difficulty parameters from these simulated response patterns. Using a dataset of over 250,000 student responses to mathematics questions, our model achieves a Pearson correlation of approximately 0.78 between predicted and actual difficulty parameters on completely unseen questions.

Related papers

• Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors [58.65143578052761]
  This paper presents a new method, MISTAKE, that constructs high-quality synthetic examples of reasoning errors.<n>We evaluate MISTAKE on three educational tasks and find that it results in (1) higher accuracy when simulating incorrect student answers.
  arXiv  Detail & Related papers  (2025-10-13T15:10:38Z)
• SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction [38.7828715471869]
  We present SMART (Simulated Students Aligned with IRT), a novel method for aligning simulated students with instructed ability.<n>We show that SMART outperforms other item difficulty prediction methods by leveraging its improved ability alignment.
  arXiv  Detail & Related papers  (2025-07-07T15:41:38Z)
• Unlocking the Potential of Difficulty Prior in RL-based Multimodal Reasoning [69.64809103333839]
  We investigate how explicitly modeling problem's difficulty prior information shapes the effectiveness of reinforcement learning based fine-tuning for multimodal reasoning.<n>Our approach demonstrates significant performances across various multi-modal mathematical reasoning benchmarks with only 2K+0.6K two-stage training data.
  arXiv  Detail & Related papers  (2025-05-19T15:43:10Z)
• Exploring the Potential of Large Language Models for Estimating the Reading Comprehension Question Difficulty [2.335292678914151]
  This study investigates the effectiveness of Large Language Models (LLMs) in estimating the difficulty of reading comprehension questions.<n>We use OpenAI's GPT-4o and o1, in estimating the difficulty of reading comprehension questions using the Study Aid and Reading Assessment (SARA) dataset.<n>The results indicate that while the models yield difficulty estimates that align meaningfully with derived IRT parameters, there are notable differences in their sensitivity to extreme item characteristics.
  arXiv  Detail & Related papers  (2025-02-25T02:28:48Z)
• Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimation [12.638577140117702]
  We show that uncertainty features contribute substantially to difficulty prediction, where difficulty is inversely proportional to the number of students who can correctly answer a question.<n>In addition to showing the value of our approach, we also observe that our model achieves state-of-the-art results on the USMLE and CMCQRD publicly available datasets.
  arXiv  Detail & Related papers  (2024-12-16T14:55:09Z)
• LLM-based Cognitive Models of Students with Misconceptions [55.29525439159345]
  This paper investigates whether Large Language Models (LLMs) can be instruction-tuned to meet this dual requirement. We introduce MalAlgoPy, a novel Python library that generates datasets reflecting authentic student solution patterns. Our insights enhance our understanding of AI-based student models and pave the way for effective adaptive learning systems.
  arXiv  Detail & Related papers  (2024-10-16T06:51:09Z)
• Can Model Uncertainty Function as a Proxy for Multiple-Choice Question Item Difficulty? [12.638577140117702]
  We leverage an aspect of generative large models which might be seen as a weakness when answering questions.<n>We explore correlations between two different metrics of uncertainty, and the actual student response distribution.
  arXiv  Detail & Related papers  (2024-07-07T10:48:04Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making.<n>We present a process-based benchmark MR-Ben that demands a meta-reasoning skill.<n>Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• Scalable Learning of Item Response Theory Models [48.91265296134559]
  Item Response Theory (IRT) models aim to assess latent abilities of $n$ examinees along with latent difficulty characteristics of $m$ test items from categorical data. We leverage the similarity of these models to logistic regression, which can be approximated accurately using small weighted subsets called coresets.
  arXiv  Detail & Related papers  (2024-03-01T17:12:53Z)
• Towards a Holistic Understanding of Mathematical Questions with Contrastive Pre-training [65.10741459705739]
  We propose a novel contrastive pre-training approach for mathematical question representations, namely QuesCo. We first design two-level question augmentations, including content-level and structure-level, which generate literally diverse question pairs with similar purposes. Then, to fully exploit hierarchical information of knowledge concepts, we propose a knowledge hierarchy-aware rank strategy.
  arXiv  Detail & Related papers  (2023-01-18T14:23:29Z)
• Learning with Instance Bundles for Reading Comprehension [61.823444215188296]
  We introduce new supervision techniques that compare question-answer scores across multiple related instances. Specifically, we normalize these scores across various neighborhoods of closely contrasting questions and/or answers. We empirically demonstrate the effectiveness of training with instance bundles on two datasets.
  arXiv  Detail & Related papers  (2021-04-18T06:17:54Z)
• Introducing a framework to assess newly created questions with Natural Language Processing [3.364554138758565]
  We propose a framework to train and evaluate models for estimating the difficulty and discrimination of newly created Multiple Choice Questions. We implement one model using this framework and test it on a real-world dataset provided by CloudAcademy.
  arXiv  Detail & Related papers  (2020-04-28T13:57:21Z)
• R2DE: a NLP approach to estimating IRT parameters of newly generated questions [3.364554138758565]
  R2DE is a model capable of assessing newly generated multiple-choice questions by looking at the text of the question. In particular, it can estimate the difficulty and the discrimination of each question.
  arXiv  Detail & Related papers  (2020-01-21T14:31:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.