Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability

• URL: http://arxiv.org/abs/2606.15029v1
• Date: Fri, 12 Jun 2026 23:54:16 GMT
• Title: Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability
• Abstract summary: We develop a method for estimating correlation-based reliability metrics of LLM judges from limited annotations.<n>Metric Match selects a subset of samples for human annotation such that the subset matches the population reliability metric.<n>We empirically show that Metric Match achieves a win-rate of 0.838 against random subset selection.
• Score: 28.15461581779256
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM judges are used to reduce the need for costly human labor in evaluating open-ended text generation. However, the reliability of these judges depends critically on their alignment with human raters -- a property that itself depends on costly human annotations. In this work, we develop a method (Metric Match) for estimating correlation-based reliability metrics of LLM judges from limited annotations. Metric Match selects a subset of samples for human annotation such that the subset matches the population reliability metric with respect to acquired synthetic labels. We empirically show that Metric Match achieves a win-rate of 0.838 against random subset selection across four different correlation metrics and 15 datasets, with an 18.7% decrease in average estimation error and reduces annotation needs by 32.5%. We provide a cost model and highlight a medical case study where our method saves $1,041.67 compared to random selection for expert annotation. Further, we shift our task from reliability estimation to reliability classification of whether a given judge is above a deployment threshold, outperforming random selection with Metric Match. All project code is publicly available, and we additionally provide an installable package for ease of use.

Related papers

• Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration [0.5294698352039444]
  We study absolute scoring tasks, which mirror more realistic use cases.<n>We show that a model's task accuracy strongly predicts its judging accuracy.<n>More capable examinee models consistently receive more lenient judgments from all judges.
  arXiv  Detail & Related papers  (2026-09-10T00:25:39Z)
• From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation [3.5637034560362935]
  We quantify the resulting judge-human disagreement at two complementary levels.<n>At the local level, we estimate per-battle uncertainty from the judge's own score differences.<n>This alone provides a drastic improvement to Elo estimation accuracy.
  arXiv  Detail & Related papers  (2026-06-11T11:38:05Z)
• Instance-Optimal Estimation with Multiple LLM Judges on a Budget [84.31744861038106]
  We formalize this question as *budgeted heteroskedastic multi-judge estimation*.<n>Given $K$ prompt-response pairs, $J$ judges with known costs, and unknown query-judge variances, the goal is to estimate a bounded score vector while minimizing an $ell_p$-error.<n>We prove that EST-IVWE matches the oracle IVWE rate up to lower-order terms in the budget.
  arXiv  Detail & Related papers  (2026-05-22T08:26:08Z)
• Distribution-Calibrated Inference time compute for Thinking LLM-as-a-Judge [5.855996386998925]
  Thinking Large Language Models (LLMs) used as judges for pairwise preferences remain noisy at the single-sample level.<n>We study inference-time compute (ITC) for evaluators that generate n independent thinking-rating samples per item.
  arXiv  Detail & Related papers  (2025-12-02T18:46:47Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs [7.197702136906138]
  We propose an uncertainty-aware fairness metric, UCerF, to enable a fine-grained evaluation of model fairness.<n> observing data size, diversity, and clarity issues in current datasets, we introduce a new gender-occupation fairness evaluation dataset.<n>We establish a benchmark, using our metric and dataset, and apply it to evaluate the behavior of ten open-source AI systems.
  arXiv  Detail & Related papers  (2025-05-29T20:45:18Z)
• Claim Extraction for Fact-Checking: Data, Models, and Automated Metrics [0.0]
  We release the FEVERFact dataset, with 17K atomic factual claims extracted from 4K contextualised Wikipedia sentences.<n>For each metric, we implement a scale using a reduction to an already-explored NLP task.<n>We validate our metrics against human grading of generic claims, to see that the model ranking on $F_fact$, our hardest metric, did not change.
  arXiv  Detail & Related papers  (2025-02-07T14:20:45Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• Estimation of Fair Ranking Metrics with Incomplete Judgments [70.37717864975387]
  We propose a sampling strategy and estimation technique for four fair ranking metrics. We formulate a robust and unbiased estimator which can operate even with very limited number of labeled items.
  arXiv  Detail & Related papers  (2021-08-11T10:57:00Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.