evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations

• URL: http://arxiv.org/abs/2607.04429v1
• Date: Sun, 05 Jul 2026 17:48:22 GMT
• Title: evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations
• Abstract summary: The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better.<n>We present evalci, a pure-Python library that turns a per-item results table into a publication-ready claim.
• Score: 0.5729426778193397
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run by more than the reported gap between models, this practice routinely overstates confidence in headline claims. The statistical machinery to fix this -- confidence intervals, paired significance tests, power analysis, clustered standard errors, multiple-comparison correction -- is well established, but no standard, pip-installable tool packages it in the shape an evaluation actually takes: a per-item results table. We present evalci, a pure-Python library (numpy/scipy/pandas only) that turns a per-item results table into a publication-ready claim -- e.g., "Model A beats Model B, $Δ=3.1$ pts, 95% CI [1.2, 5.0], paired permutation $p=0.002$, $n=1{,}319$" -- in one function call, with adapters for lm-evaluation-harness and HELM output. Every routine is validated against an independent reference (statsmodels, or brute-force exact enumeration) rather than only against itself. As a case study, we re-analyze a public comparison of nine language models' MMLU accuracy and find that 3 of the 8 adjacent leaderboard-rank gaps are not statistically significant after correcting for the 36 pairwise comparisons the ranking implies. evalci is available at https://pypi.org/project/evalci/ (source: https://github.com/Shreyaskc/evalci, DOI: https://doi.org/10.5281/zenodo.21201815)

Related papers

• Benchmarking Language Models for Statistical Problem Formulation [15.739640186203474]
  StatFormBench is a benchmark built from five cross-domain statistics textbooks and a data science case library.<n>It contains 1,013 samples spanning 20 coarse-grained and 85 fine-grained statistical problem categories.<n>No model performs consistently best across the two subtasks, while enhanced prompting strategies yield only limited or inconsistent gains.
  arXiv  Detail & Related papers  (2026-09-02T01:37:17Z)
• CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion [65.53153291902454]
  Collaborative Evaluation (CollabEval) is a simple, effective, and principled method for exploiting dependencies between historical runs of different models on the same tasks to improve statistical efficiency.<n>We build a low-rank approximation of the score matrix, and use the reconstructed values to guarantee unbiased estimates of the true evaluation metric mean.<n>We find that CollabEval substantially reduces the mean confidence interval size, and the mean squared error of the point estimate, compared to baseline methods at the same annotation budget.
  arXiv  Detail & Related papers  (2026-07-06T13:22:04Z)
• Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score [14.579552536669217]
  Unlearning in Vision-Language Models (VLMs) is required for compliance with the General Data Protection Regulation Code.<n>We present the first study of reliability in multimodal unlearning.<n>We introduce the Unified Quality Score (UQS), a metric with weights derived from each metric's Spearman correlation with the metric distance.
  arXiv  Detail & Related papers  (2026-05-04T04:13:00Z)
• Model Equality Testing: Which Model Is This API Serving? [59.005869726179455]
  API providers may quantize, watermark, or finetune the underlying model, changing the output distribution.<n>We formalize detecting such distortions by Model Equality Testing, a two-sample testing problem.<n>A test built on a simple string kernel achieves a median of 77.4% power against a range of distortions.
  arXiv  Detail & Related papers  (2024-10-26T18:34:53Z)
• Language models scale reliably with over-training and on downstream tasks [121.69867718185125]
  Scaling laws are useful guides for derisking expensive training runs. However, there remain gaps between current studies and how language models are trained. In contrast, scaling laws mostly predict loss on inference, but models are usually compared on downstream task performance.
  arXiv  Detail & Related papers  (2024-03-13T13:54:00Z)
• Paloma: A Benchmark for Evaluating Language Model Fit [112.481957296585]
  Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training.<n>We introduce Perplexity Analysis for Language Model Assessment (Paloma), a benchmark to measure LM fit to 546 English and code domains.
  arXiv  Detail & Related papers  (2023-12-16T19:12:45Z)
• Uncertainty Quantification of MLE for Entity Ranking with Covariates [3.2839905453386162]
  This paper concerns with statistical estimation and inference for the ranking problems based on pairwise comparisons. We propose a novel model, Co-Assisted Ranking Estimation (CARE) model, that extends the well-known Bradley-Terry-Luce (BTL) model. We derive the maximum likelihood estimator of $alpha_i*_i=1n$ and $beta*$ under a sparse comparison graph. We validate our theoretical results through large-scale numerical studies and an application to the mutual fund stock holding dataset.
  arXiv  Detail & Related papers  (2022-12-20T02:28:27Z)
• Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference [72.61732440246954]
  Large pre-trained language models often lack logical consistency across test inputs. We propose a framework, ConCoRD, for boosting the consistency and accuracy of pre-trained NLP models. We show that ConCoRD consistently boosts accuracy and consistency of off-the-shelf closed-book QA and VQA models.
  arXiv  Detail & Related papers  (2022-11-21T21:58:30Z)
• The Projected Covariance Measure for assumption-lean variable significance testing [3.8936058127056357]
  A simple but common approach is to specify a linear model, and then test whether the regression coefficient for $X$ is non-zero. We study the problem of testing the model-free null of conditional mean independence, i.e. that the conditional mean of $Y$ given $X$ and $Z$ does not depend on $X$. We propose a simple and general framework that can leverage flexible nonparametric or machine learning methods, such as additive models or random forests.
  arXiv  Detail & Related papers  (2022-11-03T17:55:50Z)
• Flexible Model Aggregation for Quantile Regression [92.63075261170302]
  Quantile regression is a fundamental problem in statistical learning motivated by a need to quantify uncertainty in predictions. We investigate methods for aggregating any number of conditional quantile models. All of the models we consider in this paper can be fit using modern deep learning toolkits.
  arXiv  Detail & Related papers  (2021-02-26T23:21:16Z)
• Overcoming Statistical Shortcuts for Open-ended Visual Counting [54.858754825838865]
  We aim to develop models that learn a proper mechanism of counting regardless of the output label. First, we propose the Modifying Count Distribution protocol, which penalizes models that over-rely on statistical shortcuts. Secondly, we introduce the Spatial Counting Network (SCN), which is dedicated to visual analysis and counting based on natural language questions.
  arXiv  Detail & Related papers  (2020-06-17T18:02:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.