Balanced Accuracy: The Right Metric for Evaluating LLM Judges - Explained through Youden's J statistic

• URL: http://arxiv.org/abs/2512.08121v1
• Date: Mon, 08 Dec 2025 23:58:32 GMT
• Title: Balanced Accuracy: The Right Metric for Evaluating LLM Judges - Explained through Youden's J statistic
• Abstract summary: Rigorous evaluation of large language models relies on comparing models by the prevalence of desirable or undesirable behaviors.<n>Common metrics used for this choice, such as Accuracy, Precision, and F1, are sensitive to class imbalance and to arbitrary choices of positive class.<n>We show that Balanced Accuracy is an equivalent linear transformation of $J$ and leads to better, more robust classifier selection.
• Score: 4.271443100949229
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rigorous evaluation of large language models (LLMs) relies on comparing models by the prevalence of desirable or undesirable behaviors, such as task pass rates or policy violations. These prevalence estimates are produced by a classifier, either an LLM-as-a-judge or human annotators, making the choice of classifier central to trustworthy evaluation. Common metrics used for this choice, such as Accuracy, Precision, and F1, are sensitive to class imbalance and to arbitrary choices of positive class, and can favor judges that distort prevalence estimates. We show that Youden's $J$ statistic is theoretically aligned with choosing the best judge to compare models, and that Balanced Accuracy is an equivalent linear transformation of $J$. Through both analytical arguments and empirical examples and simulations, we demonstrate how selecting judges using Balanced Accuracy leads to better, more robust classifier selection.

Related papers

• Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration [0.5294698352039444]
  We study absolute scoring tasks, which mirror more realistic use cases.<n>We show that a model's task accuracy strongly predicts its judging accuracy.<n>More capable examinee models consistently receive more lenient judgments from all judges.
  arXiv  Detail & Related papers  (2026-09-10T00:25:39Z)
• CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging [95.02210956333374]
  We propose a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation.<n>On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy.
  arXiv  Detail & Related papers  (2026-05-19T10:59:19Z)
• Bias and Uncertainty in LLM-as-a-Judge Estimation [0.0]
  LLM-as-a-Judge evaluation has become a standard tool for assessing base model performance.<n>Recent work has proposed estimators to correct this bias, but their reliability depends critically on judge quality and, for model comparisons, on calibration stability.<n>We study these failure modes through analytical results, simulations over judge quality and cross-model calibration instability.
  arXiv  Detail & Related papers  (2026-05-07T20:55:21Z)
• IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation [85.56193980646981]
  We propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following.<n>For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses.<n>Experiments on IF-RewardBench reveal significant deficiencies in current judge models.
  arXiv  Detail & Related papers  (2026-03-05T02:21:17Z)
• CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation [6.3121191919394475]
  This work introduces a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components.<n>Based on this analysis, CyclicJudge, a round-robin assignment of judges, is demonstrated to be the optimal allocation strategy.
  arXiv  Detail & Related papers  (2026-03-02T13:46:32Z)
• Who can we trust? LLM-as-a-jury for Comparative Assessment [42.32900791516691]
  Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment.<n>LLMs judges vary substantially in performance across tasks and aspects, and their judgment probabilities may be biased and inconsistent.<n>We propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone.
  arXiv  Detail & Related papers  (2026-02-18T17:04:02Z)
• Efficient Inference for Noisy LLM-as-a-Judge Evaluation [8.2511120576505]
  Large language models (LLMs) are increasingly used as automatic evaluators of generative AI outputs.<n>In practice, LLM judges are imperfect predictions for the underlying truth and can exhibit systematic, non-random errors.
  arXiv  Detail & Related papers  (2026-01-08T22:46:26Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• EigenBench: A Comparative Behavioral Measure of Value Alignment [0.28707625120094377]
  EigenBench is a black-box method for benchmarking language models' values.<n>It is designed to quantify subjective traits for which reasonable judges may disagree on the correct label.<n>It can recover model rankings on the GPQA benchmark without access to objective labels.
  arXiv  Detail & Related papers  (2025-09-02T04:14:26Z)
• Quantitative LLM Judges [60.773734899532336]
  We propose quantitative LLM judges, which align evaluation scores of existing LLM judges to humans in a given domain.<n>The models are trained to improve the score of the original judge using its rationale and score.<n>Our experiments show that quantitative judges can improve the predictive power of existing judges through post-hoc modeling.
  arXiv  Detail & Related papers  (2025-06-03T14:44:23Z)
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation [57.380464382910375]
  We show that the choice of feedback protocol for evaluation can significantly affect evaluation reliability and induce systematic biases.<n>We find that generator models can flip preferences by embedding distractor features.<n>We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.
  arXiv  Detail & Related papers  (2025-04-20T19:05:59Z)
• CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges [21.580762639442913]
  We introduce CalibraEval, a novel label-free method for mitigating selection bias during inference. CalibraEval reformulates debiasing as an optimization task aimed at adjusting observed prediction distributions to align with unbiased prediction distributions. We show that CalibraEval effectively mitigates selection bias and improves performance compared to existing debiasing methods.
  arXiv  Detail & Related papers  (2024-10-20T13:47:39Z)
• Self-rationalization improves LLM as a fine-grained judge [21.917301609125417]
  We introduce Self-Rationalization, an iterative process of improving the rationales for the judge models. Self-rationalization works by having the model generate multiple judgments with rationales for the same input. We show that our model learns to produce higher quality rationales, with a win rate of $62%$ on average compared to models just trained via SFT on rationale.
  arXiv  Detail & Related papers  (2024-10-07T21:05:53Z)
• Direct Judgement Preference Optimization [79.54459973726405]
  We train large language models (LLMs) as generative judges to evaluate and critique other models' outputs.<n>We employ three approaches to collect the preference pairs for different use cases, each aimed at improving our generative judge from a different perspective.<n>Our model robustly counters inherent biases such as position and length bias, flexibly adapts to any evaluation protocol specified by practitioners, and provides helpful language feedback for improving downstream generator models.
  arXiv  Detail & Related papers  (2024-09-23T02:08:20Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)
• A Skew-Sensitive Evaluation Framework for Imbalanced Data Classification [11.125446871030734]
  Class distribution skews in imbalanced datasets may lead to models with prediction bias towards majority classes. We propose a simple and general-purpose evaluation framework for imbalanced data classification that is sensitive to arbitrary skews in class cardinalities and importances.
  arXiv  Detail & Related papers  (2020-10-12T19:47:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.