Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficients

• URL: http://arxiv.org/abs/2603.24999v2
• Date: Fri, 27 Mar 2026 16:02:55 GMT
• Title: Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficients
• Abstract summary: We introduce a new family of nonparametric scalability coefficients based on interitem isotonic regression.<n>The central contribution is the signed isotonic $R2$, which measures the maximal proportion of variance in one item explainable.<n>We show that the signed isotonic $R2$ is extremal among monotone predictors.
• Score: 3.058685580689604
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The validity of assessments, from large-scale AI benchmarks to human classrooms, depends on the quality of individual items, yet modern evaluation instruments often contain thousands of items with minimal psychometric vetting. We introduce a new family of nonparametric scalability coefficients based on interitem isotonic regression for efficiently detecting globally bad items (e.g., miskeyed, ambiguously worded, or construct-misaligned). The central contribution is the signed isotonic $R^2$, which measures the maximal proportion of variance in one item explainable by a monotone function of another while preserving the direction of association via Kendall's $τ$. Aggregating these pairwise coefficients yields item-level scores that sharply separate problematic items from acceptable ones without assuming linearity or committing to a parametric item response model. We show that the signed isotonic $R^2$ is extremal among monotone predictors (it extracts the strongest possible monotone signal between any two items) and show that this optimality property translates directly into practical screening power. Across three AI benchmark datasets (HS Math, GSM8K, MMLU) and two human assessment datasets, the signed isotonic $R^2$ consistently achieves top-tier AUC for ranking bad items above good ones, outperforming or matching a comprehensive battery of classical test theory, item response theory, and dimensionality-based diagnostics. Crucially, the method remains robust under the small-n/large-p conditions typical of AI evaluation, requires only bivariate monotone fits computable in seconds, and handles mixed item types (binary, ordinal, continuous) without modification. It is a lightweight, model-agnostic filter that can materially reduce the reviewer effort needed to find flawed items in modern large-scale evaluation regimes.

Related papers

• Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions [51.50375419691955]
  Distributionally robust Markov decision processes provide a principled framework for sequential decision making under model uncertainty.<n>We study how many samples are necessary and sufficient to learn an $varepsilon$-optimal robust policy under the average-reward criterion.
  arXiv  Detail & Related papers  (2026-08-06T19:49:48Z)
• CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion [65.53153291902454]
  Collaborative Evaluation (CollabEval) is a simple, effective, and principled method for exploiting dependencies between historical runs of different models on the same tasks to improve statistical efficiency.<n>We build a low-rank approximation of the score matrix, and use the reconstructed values to guarantee unbiased estimates of the true evaluation metric mean.<n>We find that CollabEval substantially reduces the mean confidence interval size, and the mean squared error of the point estimate, compared to baseline methods at the same annotation budget.
  arXiv  Detail & Related papers  (2026-07-06T13:22:04Z)
• When Metrics Disagree: A Meta-Analysis of Knowledge-Graph-Completion Model Benchmarking [0.0]
  We present a meta-analysis of seven aggregators across five tests: consistency, cross-dataset stability, metric independence, robustness under noise, and generalizability.<n>Across tail $(h,r,?)$ and relation $(h,?,t)$ prediction, Z-score is the most balanced aggregator, which ranks DualE highest for tail prediction and FMS highest for relation prediction.<n>The framework resolves evaluation inconsistencies and offers evidence-based guidance for aggregator selection and model benchmarking in KGC.
  arXiv  Detail & Related papers  (2026-06-09T01:20:43Z)
• Maximum Matching Accuracy: An Instance Segmentation Evaluation Metric Utilizing Globally Optimal Matching [39.3098730337656]
  We propose a threshold-free continuous metric that finds a globally optimal one-to-one matching between predicted and ground truth objects.<n>MMA produces scores that are more stable, more sensitive, and more interpretable than existing alternatives.
  arXiv  Detail & Related papers  (2026-06-08T19:36:28Z)
• Almost Asymptotically Optimal Active Clustering Through Pairwise Observations [59.20614082241528]
  We propose a new analysis framework for clustering $M$ items into an unknown number of $K$ distinct groups using noisy and actively collected responses.<n>We establish a fundamental lower bound on the expected number of queries needed to achieve a desired confidence in the accuracy of the clustering.<n>We develop a computationally feasible variant of the Generalized Likelihood Ratio statistic and show that its performance gap to the lower bound can be accurately empirically estimated.
  arXiv  Detail & Related papers  (2026-02-05T14:16:47Z)
• RANSAC Scoring Functions: Analysis and Reality Check [0.0]
  We revisit the problem of assigning a score (a quality of fit) to candidate geometric models.<n>We show that a threshold-based parameterization leads to a unified view of likelihood-based and robust M-estimators.
  arXiv  Detail & Related papers  (2025-12-22T20:08:46Z)
• DISCO: Diversifying Sample Condensation for Efficient Model Evaluation [59.01400190971061]
  Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact.<n>We argue that promoting diversity among samples is not essential; what matters is to select samples thatmaximise diversity in model responses.<n>Our method, $textbfDiversifying Sample Condensation (DISCO)$, selects the top-k samples with the greatest model disagreements.
  arXiv  Detail & Related papers  (2025-10-09T08:53:59Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• Forest vs Tree: The $(N, K)$ Trade-off in Reproducible ML Evaluation [5.506095201822833]
  We investigate the trade-off between the number of items ($N$) and the number of responses per item ($K$) needed for reliable machine learning evaluation.<n>We find that accounting for human disagreement may come with $N times K$ at no more than 1000 for every dataset tested on at least one metric.
  arXiv  Detail & Related papers  (2025-08-05T17:18:34Z)
• What should an AI assessor optimise for? [57.96463917842822]
  An AI assessor is an external, ideally indepen-dent system that predicts an indicator, e.g., a loss value, of another AI system.<n>Here we address the question: is it always optimal to train the assessor for the target metric?<n>We experimentally explore this question for, respectively, regression losses and classification scores with monotonic and non-monotonic mappings.
  arXiv  Detail & Related papers  (2025-02-01T08:41:57Z)
• How Many Ratings per Item are Necessary for Reliable Significance Testing? [7.422152765037947]
  A cornerstone of machine learning evaluation is the assumption that model and human responses are reliable enough to evaluate models against unitary, authoritative, gold standard'' data.<n>We adapt a method to determine whether an (existing or planned) dataset has enough responses per item to assure reliable null hypothesis statistical testing.<n>We show how our methods can help AI researchers make better decisions about how to collect data for AI evaluation.
  arXiv  Detail & Related papers  (2024-12-04T02:31:28Z)
• SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluation [75.56845750400116]
  Disaggregated evaluation -- estimation of performance of a machine learning model on different subpopulations -- is a core task when assessing performance and group-fairness of AI systems. We develop SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations of blackbox models. Our method combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE)
  arXiv  Detail & Related papers  (2024-11-14T17:53:35Z)
• SUnAA: Sparse Unmixing using Archetypal Analysis [62.997667081978825]
  This paper introduces a new geological error map technique using archetypal sparse analysis (SUnAA) First, we design a new model based on archetypal sparse analysis (SUnAA)
  arXiv  Detail & Related papers  (2023-08-09T07:58:33Z)
• BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics [70.52570641514146]
  We present a benchmark of unfaithful minimal pairs (BUMP) BUMP is a dataset of 889 human-written, minimally different summary pairs. Unlike non-pair-based datasets, BUMP can be used to measure the consistency of metrics.
  arXiv  Detail & Related papers  (2022-12-20T02:17:30Z)
• Deletion and Insertion Tests in Regression Models [1.2891210250935148]
  A basic task in explainable AI (XAI) is to identify the most important features behind a prediction made by a black box function $f$. The insertion and deletion tests of Petsiuk et al. Kernel can be used to judge the quality of algorithms that rank pixels from most to at least important for a classification.
  arXiv  Detail & Related papers  (2022-05-25T00:55:47Z)
• Sparse network asymptotics for logistic regression [0.0]
  Asymptotic normality of the logistic regression is shown using a martingale central limit theorem (CLT) for triangular arrays. Sparse networks may lead to better inference since they suggest variances which incorporate additional sources of sampling variation and (ii) are valid under varying degrees of dyadic dependence.
  arXiv  Detail & Related papers  (2020-10-09T17:46:29Z)
• Ranking a set of objects: a graph based least-square approach [70.7866286425868]
  We consider the problem of ranking $N$ objects starting from a set of noisy pairwise comparisons provided by a crowd of equal workers. We propose a class of non-adaptive ranking algorithms that rely on a least-squares intrinsic optimization criterion for the estimation of qualities.
  arXiv  Detail & Related papers  (2020-02-26T16:19:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.