Judging with Confidence: Calibrating Autoraters to Preference Distributions

• URL: http://arxiv.org/abs/2510.00263v1
• Date: Tue, 30 Sep 2025 20:36:41 GMT
• Title: Judging with Confidence: Calibrating Autoraters to Preference Distributions
• Authors: Zhuohang Li, Xiaowei Li, Chengyu Huang, Guowang Li, Katayoon Goshvadi, Bo Dai, Dale Schuurmans, Paul Zhou, Hamid Palangi, Yiwen Song, Palash Goyal, Murat Kantarcioglu, Bradley A. Malin, Yuan Xue,
• Abstract summary: We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population.<n>We present two learning methods tailored to different data conditions.<n>Our results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution.
• Score: 56.17041629492863
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The alignment of large language models (LLMs) with human values increasingly relies on using other LLMs as automated judges, or ``autoraters''. However, their reliability is limited by a foundational issue: they are trained on discrete preference labels, forcing a single ground truth onto tasks that are often subjective, ambiguous, or nuanced. We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population. In this paper, we propose a general framework for calibrating probabilistic autoraters to any given preference distribution. We formalize the problem and present two learning methods tailored to different data conditions: 1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a reinforcement learning approach for sparse, binary labels. Our empirical results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution, with improved calibration and significantly lower positional bias, all while preserving performance on objective tasks.

Related papers

• Believe Your Model: Distribution-Guided Confidence Calibration [21.158022185490424]
  Internal model signals like confidence scores can partly indicate response correctness and exhibit a distributional correlation with accuracy.<n>We propose DistriVoting, which incorporates distributional priors as another signal alongside confidence during voting.<n>Our method significantly outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-03-04T09:25:36Z)
• On Optimal Steering to Achieve Exact Fairness [29.589891801235083]
  Empirically, our optimal steering techniques on both synthetic and real-world datasets improve fairness without diminishing utility.<n>We demonstrate affine steering of LLM representations to reduce bias in multi-class classification.
  arXiv  Detail & Related papers  (2025-09-19T08:37:51Z)
• SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models [0.27309692684728604]
  Large Language Models (LLMs) can achieve inflated scores on multiple-choice tasks by exploiting inherent biases in option positions or labels.<n>This study introduces SCOPE, an evaluation framework designed to measure and mitigate such selection bias in a dataset-independent manner.
  arXiv  Detail & Related papers  (2025-07-24T08:28:17Z)
• Identifying and Mitigating Social Bias Knowledge in Language Models [52.52955281662332]
  We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases.<n>FAST surpasses state-of-the-art baselines with superior debiasing performance.<n>This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
  arXiv  Detail & Related papers  (2024-08-07T17:14:58Z)
• Predicting generalization performance with correctness discriminators [56.948321040125734]
  We present a novel model that establishes upper and lower bounds on the accuracy, without requiring gold labels for the unseen data.<n>We show across a variety of tagging, parsing, and semantic parsing tasks that the gold accuracy is reliably between the predicted upper and lower bounds.
  arXiv  Detail & Related papers  (2023-11-15T22:43:42Z)
• Probabilistic Test-Time Generalization by Variational Neighbor-Labeling [62.158807685159736]
  This paper strives for domain generalization, where models are trained exclusively on source domains before being deployed on unseen target domains. Probability pseudo-labeling of target samples to generalize the source-trained model to the target domain at test time. Variational neighbor labels that incorporate the information of neighboring target samples to generate more robust pseudo labels.
  arXiv  Detail & Related papers  (2023-07-08T18:58:08Z)
• When Does Confidence-Based Cascade Deferral Suffice? [69.28314307469381]
  Cascades are a classical strategy to enable inference cost to vary adaptively across samples. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. Despite being oblivious to the structure of the cascade, confidence-based deferral often works remarkably well in practice.
  arXiv  Detail & Related papers  (2023-07-06T04:13:57Z)
• Calibrated Selective Classification [34.08454890436067]
  We develop a new approach to selective classification in which we propose a method for rejecting examples with "uncertain" uncertainties. We present a framework for learning selectively calibrated models, where a separate selector network is trained to improve the selective calibration error of a given base model. We demonstrate the empirical effectiveness of our approach on multiple image classification and lung cancer risk assessment tasks.
  arXiv  Detail & Related papers  (2022-08-25T13:31:09Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Estimating Generalization under Distribution Shifts via Domain-Invariant Representations [75.74928159249225]
  We use a set of domain-invariant predictors as a proxy for the unknown, true target labels. The error of the resulting risk estimate depends on the target risk of the proxy model.
  arXiv  Detail & Related papers  (2020-07-06T17:21:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.