Ordinality in Discrete-level Question Difficulty Estimation: Introducing Balanced DRPS and OrderedLogitNN

• URL: http://arxiv.org/abs/2507.00736v2
• Date: Thu, 03 Jul 2025 11:23:04 GMT
• Title: Ordinality in Discrete-level Question Difficulty Estimation: Introducing Balanced DRPS and OrderedLogitNN
• Authors: Arthur Thuy, Ekaterina Loginova, Dries F. Benoit,
• Abstract summary: Question difficulty is often represented using discrete levels, framing the task as ordinal regression.<n>This study addresses these limitations by benchmarking three types of model outputs.<n>We propose OrderedLogitNN, extending the ordered logit model from econometrics to neural networks.
• Score: 1.0514231683620516
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent years have seen growing interest in Question Difficulty Estimation (QDE) using natural language processing techniques. Question difficulty is often represented using discrete levels, framing the task as ordinal regression due to the inherent ordering from easiest to hardest. However, the literature has neglected the ordinal nature of the task, relying on classification or discretized regression models, with specialized ordinal regression methods remaining unexplored. Furthermore, evaluation metrics are tightly coupled to the modeling paradigm, hindering cross-study comparability. While some metrics fail to account for the ordinal structure of difficulty levels, none adequately address class imbalance, resulting in biased performance assessments. This study addresses these limitations by benchmarking three types of model outputs -- discretized regression, classification, and ordinal regression -- using the balanced Discrete Ranked Probability Score (DRPS), a novel metric that jointly captures ordinality and class imbalance. In addition to using popular ordinal regression methods, we propose OrderedLogitNN, extending the ordered logit model from econometrics to neural networks. We fine-tune BERT on the RACE++ and ARC datasets and find that OrderedLogitNN performs considerably better on complex tasks. The balanced DRPS offers a robust and fair evaluation metric for discrete-level QDE, providing a principled foundation for future research.

Related papers

• Parameterized Diffusion Optimization enabled Autoregressive Ordinal Regression for Diabetic Retinopathy Grading [53.11883409422728]
  This work proposes a novel autoregressive ordinal regression method called AOR-DR.<n>We decompose the diabetic retinopathy grading task into a series of ordered steps by fusing the prediction of the previous steps with extracted image features.<n>We exploit the diffusion process to facilitate conditional probability modeling, enabling the direct use of continuous global image features for autoregression.
  arXiv  Detail & Related papers  (2025-07-07T13:22:35Z)
• FILM: Framework for Imbalanced Learning Machines based on a new unbiased performance measure and a new ensemble-based technique [37.94431794242543]
  This research addresses the challenges of handling unbalanced datasets for binary classification tasks.<n>Standard evaluation metrics are often biased by the disproportionate representation of the minority class.<n>We propose a novel metric, the Unbiased Integration Coefficients, which exhibits significantly reduced bias.
  arXiv  Detail & Related papers  (2025-03-06T12:15:56Z)
• CLIMB-3D: Continual Learning for Imbalanced 3D Instance Segmentation [67.36817440834251]
  We propose a unified framework for textbfCLass-incremental textbfImbalance-aware textbf3DIS.<n>Our approach achieves state-of-the-art results, surpassing prior work by up to 16.76% mAP for instance segmentation and approximately 30% mIoU for semantic segmentation.
  arXiv  Detail & Related papers  (2025-02-24T18:58:58Z)
• Calibration of Ordinal Regression Networks [1.2242167538741824]
  Deep neural networks are not well-calibrated and often produce over-confident predictions.<n>We propose a novel loss function that introduces ordinal-aware calibration.<n>It incorporates soft ordinal encoding and ordinal-aware regularization to enforce both calibration and unimodality.
  arXiv  Detail & Related papers  (2024-10-21T05:56:31Z)
• Robust Capped lp-Norm Support Vector Ordinal Regression [85.84718111830752]
  Ordinal regression is a specialized supervised problem where the labels show an inherent order. Support Vector Ordinal Regression, as an outstanding ordinal regression model, is widely used in many ordinal regression tasks. We introduce a new model, Capped $ell_p$-Norm Support Vector Ordinal Regression(CSVOR), that is robust to outliers.
  arXiv  Detail & Related papers  (2024-04-25T13:56:05Z)
• Conformal Prediction via Regression-as-Classification [15.746085775084238]
  We convert regression to a classification problem and then use CP for classification to obtain CP sets for regression. Empirical results on many benchmarks shows that this simple approach gives surprisingly good results on many practical problems.
  arXiv  Detail & Related papers  (2024-04-12T00:21:30Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Deep Imbalanced Regression via Hierarchical Classification Adjustment [50.19438850112964]
  Regression tasks in computer vision are often formulated into classification by quantizing the target space into classes. The majority of training samples lie in a head range of target values, while a minority of samples span a usually larger tail range. We propose to construct hierarchical classifiers for solving imbalanced regression tasks. Our novel hierarchical classification adjustment (HCA) for imbalanced regression shows superior results on three diverse tasks.
  arXiv  Detail & Related papers  (2023-10-26T04:54:39Z)
• Clipped Stochastic Methods for Variational Inequalities with Heavy-Tailed Noise [64.85879194013407]
  We prove the first high-probability results with logarithmic dependence on the confidence level for methods for solving monotone and structured non-monotone VIPs. Our results match the best-known ones in the light-tails case and are novel for structured non-monotone problems. In addition, we numerically validate that the gradient noise of many practical formulations is heavy-tailed and show that clipping improves the performance of SEG/SGDA.
  arXiv  Detail & Related papers  (2022-06-02T15:21:55Z)
• Ordinal-ResLogit: Interpretable Deep Residual Neural Networks for Ordered Choices [6.982614422666432]
  We develop a fully interpretable deep learning-based ordinal regression model. Formulations for market share, substitution patterns, and elasticities are derived. Our results show that Ordinal-ResLogit outperforms the traditional ordinal regression model for both datasets.
  arXiv  Detail & Related papers  (2022-04-20T02:14:28Z)
• Continuously Generalized Ordinal Regression for Linear and Deep Models [41.03778663275373]
  Ordinal regression is a classification task where classes have an order and prediction error increases the further the predicted class is from the true class. We propose a new approach for modeling ordinal data that allows class-specific hyperplane slopes. Our method significantly outperforms the standard ordinal logistic model over a thorough set of ordinal regression benchmark datasets.
  arXiv  Detail & Related papers  (2022-02-14T19:49:05Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.