Calibrated Preference Learning: The Case of Label Ranking

• URL: http://arxiv.org/abs/2605.30447v1
• Date: Thu, 28 May 2026 18:18:21 GMT
• Title: Calibrated Preference Learning: The Case of Label Ranking
• Abstract summary: We formalize calibration for label ranking and develop a hierarchy of notions covering full rankings, sub-rankings, and top-k rankings.<n>We find popular label ranking models are often poorly calibrated, with substantial differences between sub-ranking and top-k metrics.<n>Applying our framework to RLHF reward models, we find that calibration correlates strongly but not perfectly with benchmark accuracy.
• Score: 32.50954014661671
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Calibration, the alignment of predicted probabilities with true outcome frequencies, is essential for reliable decision-making. While extensively studied for classification and regression, calibration has not been formally addressed for probabilistic label ranking, where the goal is to predict a distribution over orderings of a label set. Naively treating rankings as classes ignores their structure and fails to capture important modalities such as pairwise and top-k predictions. We formalize calibration for label ranking and develop a hierarchy of notions covering full rankings, sub-rankings, and top-k rankings. We prove that full-rank calibration implies the others but not conversely, and sub-ranking and top-k calibration are incomparable. Empirically, we find popular label ranking models are often poorly calibrated, with substantial differences between sub-ranking and top-k metrics. Applying our framework to RLHF reward models, we find that calibration correlates strongly but not perfectly with benchmark accuracy, suggesting it captures a meaningful quality dimension beyond top-1 accuracy. These findings motivate future work on understanding the downstream effects of miscalibration and developing methods to correct it.

Related papers

• Scalable Utility-Aware Multiclass Calibration [53.28176049547449]
  Utility calibration is a general framework that measures the calibration error relative to a specific utility function.<n>We demonstrate how this framework can unify and re-interpret several existing calibration metrics.
  arXiv  Detail & Related papers  (2025-10-29T12:32:14Z)
• Practical estimation of the optimal classification error with soft labels and calibration [47.001801756596926]
  We extend a previous work that utilizes soft labels for estimating the Bayes error, the optimal error rate.<n>We tackle a more challenging problem setting: estimation with corrupted soft labels.<n>Our method is instance-free, i.e., we do not assume access to any input instances.
  arXiv  Detail & Related papers  (2025-05-27T06:04:57Z)
• Rethinking Early Stopping: Refine, Then Calibrate [49.966899634962374]
  We present a novel variational formulation of the calibration-refinement decomposition.<n>We provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training.
  arXiv  Detail & Related papers  (2025-01-31T15:03:54Z)
• Calibration of Ordinal Regression Networks [1.2242167538741824]
  Deep neural networks are not well-calibrated and often produce over-confident predictions.<n>We propose a novel loss function that introduces ordinal-aware calibration.<n>It incorporates soft ordinal encoding and ordinal-aware regularization to enforce both calibration and unimodality.
  arXiv  Detail & Related papers  (2024-10-21T05:56:31Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• Model Calibration in Dense Classification with Adaptive Label Perturbation [44.62722402349157]
  Existing dense binary classification models are prone to being over-confident. We propose Adaptive Label Perturbation (ASLP) which learns a unique label perturbation level for each training image. ASLP can significantly improve calibration degrees of dense binary classification models on both in-distribution and out-of-distribution data.
  arXiv  Detail & Related papers  (2023-07-25T14:40:11Z)
• Optimizing Partial Area Under the Top-k Curve: Theory and Practice [151.5072746015253]
  We develop a novel metric named partial Area Under the top-k Curve (AUTKC) AUTKC has a better discrimination ability, and its Bayes optimal score function could give a correct top-K ranking with respect to the conditional probability. We present an empirical surrogate risk minimization framework to optimize the proposed metric.
  arXiv  Detail & Related papers  (2022-09-03T11:09:13Z)
• Better Uncertainty Calibration via Proper Scores for Classification and Beyond [15.981380319863527]
  We introduce the framework of proper calibration errors, which relates every calibration error to a proper score. This relationship can be used to reliably quantify the model calibration improvement.
  arXiv  Detail & Related papers  (2022-03-15T12:46:08Z)
• Top-label calibration [3.3504365823045044]
  We study the problem of post-hoc calibration for multiclass classification, with an emphasis on histogram binning. We find that the popular notion of confidence calibration is not sufficiently strong -- there exist predictors that are not calibrated in any meaningful way but are perfectly confidence calibrated. We propose a closely related (but subtly different) notion, top-label calibration, that accurately captures the intuition and simplicity of confidence calibration, but addresses its drawbacks.
  arXiv  Detail & Related papers  (2021-07-18T03:27:50Z)
• Distribution-free uncertainty quantification for classification under label shift [105.27463615756733]
  We focus on uncertainty quantification (UQ) for classification problems via two avenues. We first argue that label shift hurts UQ, by showing degradation in coverage and calibration. We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.
  arXiv  Detail & Related papers  (2021-03-04T20:51:03Z)
• Unsupervised Calibration under Covariate Shift [92.02278658443166]
  We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
  arXiv  Detail & Related papers  (2020-06-29T21:50:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.