Auditing LLM Benchmarks with Item Response Theory

• URL: http://arxiv.org/abs/2605.30504v1
• Date: Thu, 28 May 2026 19:38:44 GMT
• Title: Auditing LLM Benchmarks with Item Response Theory
• Abstract summary: LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all.<n>We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples.<n>We trace these errors to mechanical labelings, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label.
• Score: 1.1455131351238124
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all. We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple-choice benchmarks using responses from 114 models, outperforming a supervised classifier. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label. The same model fit reveals that reward models specialize in stylistic preference rather than factual knowledge, and identifies one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with benchmark contamination or benchmark-specific over-optimization.

Related papers

• Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence [0.7340017786387767]
  We show that prior scores, even when included only as context metadata, anchor judgments and systematically shift ratings toward their values.<n>On categorical industry data with human-labeled ground truth, anchored metadata blocks 48% of error corrections and flips 10.18% of correct judgments toward an assigned wrong label.
  arXiv  Detail & Related papers  (2026-08-26T14:41:24Z)
• Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks [19.555897987977445]
  We introduce the LLM Nominal Response Model (LLM-NRM), an option-aware framework that models the full distribution over answer choices.<n>We show that incorrect answers carry distinct and useful measurement information rather than representing equivalent mistakes.
  arXiv  Detail & Related papers  (2026-08-03T23:59:39Z)
• BenchBench: Benchmarking Automated Benchmark Generation [10.44497524694021]
  BenchBench is a pipeline and dataset for benchmarking automated benchmark generation.<n>We generate 16.7K items, retain 15K core items post-filtering, and produce 152K graded model--item responses.
  arXiv  Detail & Related papers  (2026-03-21T13:05:32Z)
• Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks [26.89839484242575]
  ATLAS is an adaptive testing framework for large language models.<n>It estimates model ability through Fisher information-guided item selection.<n>It achieves 90% item reduction while maintaining measurement precision.
  arXiv  Detail & Related papers  (2025-10-26T03:54:12Z)
• Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures [87.75098311090642]
  Current preference learning methods achieve high accuracy on standard benchmarks but exhibit significant performance degradation when objective quality signals are removed.<n>We introduce WritingPreferenceBench, a dataset of 1,800 human-annotated preference pairs (1,200 English, 600 Chinese) across 8 creative writing genres.
  arXiv  Detail & Related papers  (2025-10-16T12:23:13Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Ranked from Within: Ranking Large Multimodal Models Without Labels [73.96543593298426]
  We show that uncertainty scores derived from softmax distributions provide a robust basis for ranking models across various tasks.<n>This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.
  arXiv  Detail & Related papers  (2024-12-09T13:05:43Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance [28.524573212179124]
  Large language models (LLMs) offer new opportunities to enhance the annotation process.<n>We compare expert, crowd-sourced, and LLM-based annotations in terms of the agreement, label quality, and efficiency.<n>Our findings reveal a substantial number of label errors, which, when corrected, a significant upward shift in reported model performance.
  arXiv  Detail & Related papers  (2024-10-24T16:27:03Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)
• MARS: Model-agnostic Biased Object Removal without Additional Supervision for Weakly-Supervised Semantic Segmentation [3.997707534484444]
  Weakly-supervised semantic segmentation aims to reduce labeling costs by training semantic segmentation models using weak supervision. We propose a fully-automatic/model-agnostic biased object removal framework called MARS. We show that MARS consistently improves the performance of various WSSS models by at least 30% without additional supervision.
  arXiv  Detail & Related papers  (2023-04-19T18:24:28Z)
• Dist-PU: Positive-Unlabeled Learning from a Label Distribution Perspective [89.5370481649529]
  We propose a label distribution perspective for PU learning in this paper. Motivated by this, we propose to pursue the label distribution consistency between predicted and ground-truth label distributions. Experiments on three benchmark datasets validate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2022-12-06T07:38:29Z)
• Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors [105.12462629663757]
  In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models.
  arXiv  Detail & Related papers  (2022-05-25T15:26:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.