Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models

• URL: http://arxiv.org/abs/2601.02580v1
• Date: Mon, 05 Jan 2026 22:11:41 GMT
• Title: Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models
• Authors: Christopher Ormerod,
• Abstract summary: This study introduces a novel approach that implicitly models Item Response Theory (IRT) psychometric properties.<n>We train models to generate responses to multiple choice questions conditioned on discrete ability descriptors.<n>We reconstruct the probability of a correct response as a function of student ability, effectively generating synthetic Item Characteristic Curves (ICCs) to estimate IRT parameters.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Traditional methods for determining assessment item parameters, such as difficulty and discrimination, rely heavily on expensive field testing to collect student performance data for Item Response Theory (IRT) calibration. This study introduces a novel approach that implicitly models these psychometric properties by fine-tuning Large Language Models (LLMs) to simulate student responses across a spectrum of latent abilities. Leveraging the Qwen-3 dense model series and Low-Rank Adaptation (LoRA), we train models to generate responses to multiple choice questions conditioned on discrete ability descriptors. We reconstruct the probability of a correct response as a function of student ability, effectively generating synthetic Item Characteristic Curves (ICCs) to estimate IRT parameters. Evaluation on a dataset of Grade 6 English Language Arts (ELA) items and the BEA 2024 Shared Task dataset demonstrates that this method competes with or outperforms baseline approaches. This simulation-based technique seems particularly effective at modeling item discrimination.

Related papers

• Learning Compact Representations of LLM Abilities via Item Response Theory [35.74367665390977]
  We explore how to learn compact representations of large language models (LLMs)<n>We frame this problem as estimating the probability that a given model will correctly answer a specific query.<n>To learn these parameters jointly, we introduce a Mixture-of-Experts (MoE) network that couples model- and query-level embeddings.
  arXiv  Detail & Related papers  (2025-10-01T12:55:34Z)
• Model Correlation Detection via Random Selection Probing [62.093777777813756]
  Existing similarity-based methods require access to model parameters or produce scores without thresholds.<n>We introduce Random Selection Probing (RSP), a hypothesis-testing framework that formulates model correlation detection as a statistical test.<n>RSP produces rigorous p-values that quantify evidence of correlation.
  arXiv  Detail & Related papers  (2025-09-29T01:40:26Z)
• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• Model-free Methods for Event History Analysis and Efficient Adjustment (PhD Thesis) [55.2480439325792]
  This thesis is a series of independent contributions to statistics unified by a model-free perspective.<n>The first chapter elaborates on how a model-free perspective can be used to formulate flexible methods that leverage prediction techniques from machine learning.<n>The second chapter studies the concept of local independence, which describes whether the evolution of one process is directly influenced by another.
  arXiv  Detail & Related papers  (2025-02-11T19:24:09Z)
• How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics [49.9329723199239]
  We propose a method for the automated creation of a challenging test set without relying on the manual construction of artificial and unrealistic examples. We categorize the test set of popular NLI datasets into three difficulty levels by leveraging methods that exploit training dynamics. When our characterization method is applied to the training set, models trained with only a fraction of the data achieve comparable performance to those trained on the full dataset.
  arXiv  Detail & Related papers  (2024-10-04T13:39:21Z)
• The Role of Model Architecture and Scale in Predicting Molecular Properties: Insights from Fine-Tuning RoBERTa, BART, and LLaMA [0.0]
  This study introduces a systematic framework to compare the efficacy of Large Language Models (LLMs) for fine-tuning across various cheminformatics tasks. We assessed three well-known models-RoBERTa, BART, and LLaMA-on their ability to predict molecular properties. We found that LLaMA-based models generally offered the lowest validation loss, suggesting their superior adaptability across tasks and scales.
  arXiv  Detail & Related papers  (2024-05-02T02:20:12Z)
• Diffusion posterior sampling for simulation-based inference in tall data settings [53.17563688225137]
  Simulation-based inference ( SBI) is capable of approximating the posterior distribution that relates input parameters to a given observation. In this work, we consider a tall data extension in which multiple observations are available to better infer the parameters of the model. We compare our method to recently proposed competing approaches on various numerical experiments and demonstrate its superiority in terms of numerical stability and computational cost.
  arXiv  Detail & Related papers  (2024-04-11T09:23:36Z)
• Latent class analysis with weighted responses [0.0]
  We propose a novel generative model, the weighted latent class model (WLCM) Our model allows data's response matrix to be generated from an arbitrary distribution with a latent class structure. We investigate the identifiability of the model and propose an efficient algorithm for estimating the latent classes and other model parameters.
  arXiv  Detail & Related papers  (2023-10-17T04:16:20Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• Multidimensional Item Response Theory in the Style of Collaborative Filtering [0.8057006406834467]
  This paper presents a machine learning approach to multidimensional item response theory (MIRT) Inspired by collaborative filtering, we define a general class of models that includes many MIRT models. We discuss the use of penalized joint maximum likelihood (JML) to estimate individual models and cross-validation to select the best performing model.
  arXiv  Detail & Related papers  (2023-01-03T00:56:27Z)
• Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation [24.65301562548798]
  We study the problem of model selection in causal inference, specifically for conditional average treatment effect (CATE) estimation. We conduct an empirical analysis to benchmark the surrogate model selection metrics introduced in the literature, as well as the novel ones introduced in this work.
  arXiv  Detail & Related papers  (2022-11-03T16:26:06Z)
• MACE: An Efficient Model-Agnostic Framework for Counterfactual Explanation [132.77005365032468]
  We propose a novel framework of Model-Agnostic Counterfactual Explanation (MACE) In our MACE approach, we propose a novel RL-based method for finding good counterfactual examples and a gradient-less descent method for improving proximity. Experiments on public datasets validate the effectiveness with better validity, sparsity and proximity.
  arXiv  Detail & Related papers  (2022-05-31T04:57:06Z)
• Calibrating Over-Parametrized Simulation Models: A Framework via Eligibility Set [3.862247454265944]
  We develop a framework to develop calibration schemes that satisfy rigorous frequentist statistical guarantees. We demonstrate our methodology on several numerical examples, including an application to calibration of a limit order book market simulator.
  arXiv  Detail & Related papers  (2021-05-27T00:59:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.