Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation

• URL: http://arxiv.org/abs/2606.08679v1
• Date: Sun, 07 Jun 2026 15:31:29 GMT
• Title: Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation
• Abstract summary: We introduce a hierarchical framework that constructs model rank intervals with statistical guarantees at both levels.<n>Experiments on simulated data and the TabArena and PromptEval benchmarks show that our method yields statistically valid and informative intervals.
• Score: 4.297070083645049
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretrained models are often evaluated on multi-task leaderboards to measure their applicability in diverse contexts. However, current methods for aggregating performance across tasks into leaderboard-level rankings do not address the uncertainty and variability at the task level. While recent works have proposed interval-based model rankings, the principled aggregation of uncertainty from individual tasks to leaderboard-level rankings remains unaddressed, and variation in models' performance across tasks is frequently obscured. In this work, we introduce a hierarchical framework that constructs model rank intervals with statistical guarantees at both levels: task-level rank confidence intervals from pairwise comparisons, and leaderboard-level rank prediction intervals using a conformal approach. This enables reliable quantification of model rank for each observed task and for new potential tasks. Experiments on simulated data and the TabArena and PromptEval (MMLU) benchmarks show that our method yields statistically valid and informative intervals, enabling reliable, uncertainty-aware model ranking on leaderboards.

Related papers

• Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons [16.73008463123115]
  Low-rank sharing improves sample efficiency over independent task-wise Bradley-Terry estimation.<n>Experiments on synthetic data and Arena show that low-rank sharing improves sample efficiency over independent task-wise Bradley-Terry estimation.
  arXiv  Detail & Related papers  (2026-05-28T05:44:43Z)
• IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation [85.56193980646981]
  We propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following.<n>For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses.<n>Experiments on IF-RewardBench reveal significant deficiencies in current judge models.
  arXiv  Detail & Related papers  (2026-03-05T02:21:17Z)
• Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification [9.99813918008511]
  We study prompt-dependent ranking inference under pairwise human preferences.<n>We develop a framework for decision-safe rankings with statistically valid uncertainty guarantees.
  arXiv  Detail & Related papers  (2026-02-11T18:16:24Z)
• OrdRankBen: A Novel Ranking Benchmark for Ordinal Relevance in NLP [6.6002656593260225]
  Benchmark datasets play a crucial role in providing standardized testbeds that ensure fair comparisons.<n>Existing NLP ranking benchmarks typically use binary relevance labels or continuous relevance scores, neglecting ordinal relevance scores.<n>We introduce OrdRankBen, a novel benchmark designed to capture multi-granularity relevance distinctions.
  arXiv  Detail & Related papers  (2025-03-02T00:28:55Z)
• Learning when to rank: Estimation of partial rankings from sparse, noisy comparisons [0.0]
  We develop a principled nonparametric Bayesian method for learning partial rankings (rankings with ties)<n>We examine the performance of our method on a variety of real and synthetic network datasets.
  arXiv  Detail & Related papers  (2025-01-05T11:04:30Z)
• Ranked from Within: Ranking Large Multimodal Models Without Labels [73.96543593298426]
  We show that uncertainty scores derived from softmax distributions provide a robust basis for ranking models across various tasks.<n>This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.
  arXiv  Detail & Related papers  (2024-12-09T13:05:43Z)
• Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment [54.179859639868646]
  We propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories.
  arXiv  Detail & Related papers  (2023-07-27T07:42:44Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)
• Document Ranking with a Pretrained Sequence-to-Sequence Model [56.44269917346376]
  We show how a sequence-to-sequence model can be trained to generate relevance labels as "target words" Our approach significantly outperforms an encoder-only model in a data-poor regime.
  arXiv  Detail & Related papers  (2020-03-14T22:29:50Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.