Analysis of Learner Independent Variables for Estimating Assessment Items Difficulty Level

• URL: http://arxiv.org/abs/2206.04416v1
• Date: Thu, 9 Jun 2022 11:01:57 GMT
• Title: Analysis of Learner Independent Variables for Estimating Assessment Items Difficulty Level
• Abstract summary: This research proposes a model for predicting the difficulty level of assessment items in engineering courses. Experimental results from three engineering courses provide around 80% accuracy in classification of items using the proposed model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The quality of assessment determines the quality of learning, and is characterized by validity, reliability and difficulty. Mastery of learning is generally represented by the difficulty levels of assessment items. A very large number of variables are identified in the literature to measure the difficulty level. These variables, which are not completely independent of one another, are categorized into learner dependent, learner independent, generic, non-generic and score based. This research proposes a model for predicting the difficulty level of assessment items in engineering courses using learner independent and generic variables. An ordinal regression model is developed for predicting the difficulty level, and uses six variables including three stimuli variables (item presentation, usage of technical notations and number of resources), two content related variables (number of concepts and procedures) and one task variable (number of conditions). Experimental results from three engineering courses provide around 80% accuracy in classification of items using the proposed model.

Related papers

• Interpreting Learning Under Competing Models: Joint and Stepwise Approaches for Dynamic Cognitive Diagnosis [0.0]
  We analyse data from two reading games measuring vocabulary and comprehension from Grade 2 to Grade 3.<n>We show that this decision can change substantive conclusions about how learners develop.<n>A simulation study identifies when the two analyses diverge and shows that joint analysis is more reliable when the item-skill structure is uncertain.
  arXiv  Detail & Related papers  (2026-06-05T01:08:48Z)
• Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling [6.372025352636228]
  We introduce a multi-level bootstrapping approach to realistically model annotator behavior.<n>We analyze the tradeoffs between the number of items ($N$) and the number of responses per item ($K$) required to achieve statistical significance.
  arXiv  Detail & Related papers  (2026-05-13T17:22:27Z)
• MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoning [0.0]
  Large language models excel on math benchmarks, but their math reasoning robustness to linguistic variation is underexplored.<n>We introduce MathRobust-LV, a test set and evaluation methodology that mirrors how instructors rephrase problems across assessments.<n>Our results highlight that robustness to linguistic variation is a fundamental challenge, exposing reasoning vulnerabilities in models.
  arXiv  Detail & Related papers  (2025-10-07T20:09:29Z)
• The Course Difficulty Analysis Cookbook [0.7373617024876725]
  Curriculum analytics (CA) studies curriculum structure and student data to ensure the quality of educational programs.<n>This paper provides the first comprehensive review and comparison of existing approaches for assessing course difficulty.<n>It offers a hands-on tutorial on model selection, assumption checking, and practical CA applications.
  arXiv  Detail & Related papers  (2025-08-17T10:55:37Z)
• Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding [53.63482987410292]
  We present a self-adaptive curriculum learning paradigm that prioritizes fine-tuning examples based on difficulty scores predicted by pre-trained language models.<n>We evaluate our method on four natural language understanding (NLU) datasets covering both binary and multi-class classification tasks.
  arXiv  Detail & Related papers  (2025-07-13T19:36:17Z)
• Exploring the Potential of Large Language Models for Estimating the Reading Comprehension Question Difficulty [2.335292678914151]
  This study investigates the effectiveness of Large Language Models (LLMs) in estimating the difficulty of reading comprehension questions. We use OpenAI's GPT-4o and o1, in estimating the difficulty of reading comprehension questions using the Study Aid and Reading Assessment (SARA) dataset. The results indicate that while the models yield difficulty estimates that align meaningfully with derived IRT parameters, there are notable differences in their sensitivity to extreme item characteristics.
  arXiv  Detail & Related papers  (2025-02-25T02:28:48Z)
• Modeling Behavior Change for Multi-model At-Risk Students Early Prediction (extended version) [10.413751893289056]
  Current models primarily identify students with consistently poor performance through simple and discrete behavioural patterns. We have developed an innovative prediction model, Multimodal- ChangePoint Detection (MCPD), utilizing the textual teacher remark data and numerical grade data from middle schools. Our model achieves an accuracy range of 70- 75%, with an average outperforming baseline algorithms by approximately 5-10%.
  arXiv  Detail & Related papers  (2025-02-19T11:16:46Z)
• Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimation [12.638577140117702]
  We show that uncertainty features contribute substantially to difficulty prediction, where difficulty is inversely proportional to the number of students who can correctly answer a question. In addition to showing the value of our approach, we also observe that our model achieves state-of-the-art results on the USMLE and CMCQRD publicly available datasets.
  arXiv  Detail & Related papers  (2024-12-16T14:55:09Z)
• Towards Understanding the Feasibility of Machine Unlearning [14.177012256360635]
  We present a set of novel metrics for quantifying the difficulty of unlearning. Specifically, we propose several metrics to assess the conditions necessary for a successful unlearning operation. We also present a ranking mechanism to identify the most challenging samples to unlearn.
  arXiv  Detail & Related papers  (2024-10-03T23:41:42Z)
• Lessons from the Trenches on Reproducible Evaluation of Language Models [60.522749986793094]
  We draw on three years of experience in evaluating large language models to provide guidance and lessons for researchers. We present the Language Model Evaluation Harness (lm-eval), an open source library for independent, reproducible, and evaluation of language models.
  arXiv  Detail & Related papers  (2024-05-23T16:50:49Z)
• DCID: Deep Canonical Information Decomposition [84.59396326810085]
  We consider the problem of identifying the signal shared between two one-dimensional target variables. We propose ICM, an evaluation metric which can be used in the presence of ground-truth labels. We also propose Deep Canonical Information Decomposition (DCID) - a simple, yet effective approach for learning the shared variables.
  arXiv  Detail & Related papers  (2023-06-27T16:59:06Z)
• A quantitative study of NLP approaches to question difficulty estimation [0.30458514384586394]
  This work quantitatively analyzes several approaches proposed in previous research, and comparing their performance on datasets from different educational domains. We find that Transformer based models are the best performing across different educational domains, with DistilBERT performing almost as well as BERT. As for the other models, the hybrid ones often outperform the ones based on a single type of features, the ones based on linguistic features perform well on reading comprehension questions, while frequency based features (TF-IDF) and word embeddings (word2vec) perform better in domain knowledge assessment.
  arXiv  Detail & Related papers  (2023-05-17T14:26:00Z)
• Determination of class-specific variables in nonparametric multiple-class classification [0.0]
  We propose a probability-based nonparametric multiple-class classification method, and integrate it with the ability of identifying high impact variables for individual class. We report the properties of the proposed method, and use both synthesized and real data sets to illustrate its properties under different classification situations.
  arXiv  Detail & Related papers  (2022-05-07T10:08:58Z)
• Analyzing the Limits of Self-Supervision in Handling Bias in Language [52.26068057260399]
  We evaluate how well language models capture the semantics of four tasks for bias: diagnosis, identification, extraction and rephrasing. Our analyses indicate that language models are capable of performing these tasks to widely varying degrees across different bias dimensions, such as gender and political affiliation.
  arXiv  Detail & Related papers  (2021-12-16T05:36:08Z)
• Conditional Contrastive Learning: Removing Undesirable Information in Self-Supervised Representations [108.29288034509305]
  We develop conditional contrastive learning to remove undesirable information in self-supervised representations. We demonstrate empirically that our methods can successfully learn self-supervised representations for downstream tasks.
  arXiv  Detail & Related papers  (2021-06-05T10:51:26Z)
• Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering [80.60605604261416]
  We propose a novel neuro-symbolic framework for zero-shot question answering across commonsense tasks. We vary the set of language models, training regimes, knowledge sources, and data generation strategies, and measure their impact across tasks. We show that, while an individual knowledge graph is better suited for specific tasks, a global knowledge graph brings consistent gains across different tasks.
  arXiv  Detail & Related papers  (2020-11-07T22:52:21Z)
• Curriculum Learning with Diversity for Supervised Computer Vision Tasks [1.5229257192293197]
  We introduce a novel curriculum sampling strategy which takes into consideration the diversity of the training data together with the difficulty of the inputs. We prove that our strategy is very efficient for unbalanced data sets, leading to faster convergence and more accurate results.
  arXiv  Detail & Related papers  (2020-09-22T15:32:49Z)
• Automatic Validation of Textual Attribute Values in E-commerce Catalog by Learning with Limited Labeled Data [61.789797281676606]
  We propose a novel meta-learning latent variable approach, called MetaBridge. It can learn transferable knowledge from a subset of categories with limited labeled data. It can capture the uncertainty of never-seen categories with unlabeled data.
  arXiv  Detail & Related papers  (2020-06-15T21:31:05Z)
• Be Like Water: Robustness to Extraneous Variables Via Adaptive Feature Normalization [17.829013101192295]
  Extraneous variables are variables that are irrelevant for a certain task, but heavily affect the distribution of the available data. We show that the presence of such variables can degrade the performance of deep-learning models. We show that estimating the feature statistics adaptively during inference, as in instance normalization, addresses this issue.
  arXiv  Detail & Related papers  (2020-02-10T18:47:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.