Confidence Calibration under Ambiguous Ground Truth

• URL: http://arxiv.org/abs/2603.22879v1
• Date: Tue, 24 Mar 2026 07:24:09 GMT
• Title: Confidence Calibration under Ambiguous Ground Truth
• Abstract summary: Post-hoc calibrators fitted on majority-voted labels can appear well-calibrated under conventional evaluation.<n>But they remain substantially miscalibrated against the underlying annotator distribution.<n>We develop a family of ambiguity-aware post-hoc calibrators that optimise proper scoring rules against the full label distribution.
• Score: 43.71398545904091
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Confidence calibration assumes a unique ground-truth label per input, yet this assumption fails wherever annotators genuinely disagree. Post-hoc calibrators fitted on majority-voted labels, the standard single-label targets used in practice, can appear well-calibrated under conventional evaluation yet remain substantially miscalibrated against the underlying annotator distribution. We show that this failure is structural: under simplifying assumptions, Temperature Scaling is biased toward temperatures that underestimate annotator uncertainty, with true-label miscalibration increasing monotonically with annotation entropy. To address this, we develop a family of ambiguity-aware post-hoc calibrators that optimise proper scoring rules against the full label distribution and require no model retraining. Our methods span progressively weaker annotation requirements: Dirichlet-Soft leverages the full annotator distribution and achieves the best overall calibration quality across settings; Monte Carlo Temperature Scaling with a single annotation per example (MCTS S=1) matches full-distribution calibration across all benchmarks, demonstrating that pre-aggregated label distributions are unnecessary; and Label-Smooth Temperature Scaling (LS-TS) operates with voted labels alone by constructing data-driven pseudo-soft targets from the model's own confidence. Experiments on four benchmarks with real multi-annotator distributions (CIFAR-10H, ChaosNLI) and clinically-informed synthetic annotations (ISIC~2019, DermaMNIST) show that Dirichlet-Soft reduces true-label ECE by 55-87% relative to Temperature Scaling, while LS-TS reduces ECE by 9-77% without any annotator data.

Related papers

• Respect Your Zero-Shot Uncertainty: Conservative Calibration for Test-Time-Adapted Vision-Language Models [5.63022899921334]
  We show that TTA can increase confidence and reduce entropy even when the top-1 prediction and its correctness remain unchanged.<n>We propose Zero-Shot-Anchored Entropy (ZAEC), a label-free post-hoc method that uses zero-shot entropy as a sample-specific uncertainty reference.
  arXiv  Detail & Related papers  (2026-08-06T12:11:53Z)
• Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributions [0.0]
  We study whether temperature scaling retains its calibration properties when this assumption is violated.<n>We find that temperature scaling on hard labels consistently underperforms an oracle directly on soft labels.<n>These findings suggest that calibration protocols built on majority-vote labels systematically misstate model reliability wherever label ambiguity is structural.
  arXiv  Detail & Related papers  (2026-07-15T03:52:06Z)
• FedLAS: Feature-Modulated Bidirectional Label Smoothing for Neural Network Calibration [2.2346611200823934]
  Deep Neural Network (DNN) classifiers suffer from poor calibration when their softmax outputs deviate from the empirical likelihoods.<n>We propose FedLAS: Feature-Modulated Bidirectional Label Smoothing, a plug-and-play algorithm for label smoothing-based losses.
  arXiv  Detail & Related papers  (2026-06-26T23:55:44Z)
• CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction [51.56484100374058]
  Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness.<n>We introduce textscCaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction.<n>textscCaliDist consistently achieves lower Expected Error (ECE) and Brier Score compared with strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T07:27:53Z)
• DiCaP: Distribution-Calibrated Pseudo-labeling for Semi-Supervised Multi-Label Learning [83.94574004953346]
  Semi-supervised multi-label learning aims to leverage unlabeled data to improve the model's performance.<n>Most existing methods assign equal weights to all pseudo-labels regardless of their quality.<n>We propose Distribution-Calibrated Pseudo-labeling (DiCaP), a correctness-aware framework that estimates posterior precision to calibrate pseudo-label weights.
  arXiv  Detail & Related papers  (2025-11-25T11:55:02Z)
• Beyond One-Hot Labels: Semantic Mixing for Model Calibration [22.39558434131574]
  We present textbfCalibration-aware Semantic Mixing (CSM), a novel framework that generates training samples with mixed class characteristics.<n>We show that CSM achieves superior calibration compared to the state-of-the-art calibration approaches.
  arXiv  Detail & Related papers  (2025-04-18T08:26:18Z)
• All Models Are Miscalibrated, But Some Less So: Comparing Calibration with Conditional Mean Operators [12.103487148356747]
  We propose a kernel calibration error based on the Hilbert-Schmidt norm of the difference between conditional mean operators.<n>Our experiments show that CKCE provides a more consistent ranking of models by their calibration error and is more robust against distribution shift.
  arXiv  Detail & Related papers  (2025-02-17T05:52:09Z)
• Model Calibration in Dense Classification with Adaptive Label Perturbation [44.62722402349157]
  Existing dense binary classification models are prone to being over-confident. We propose Adaptive Label Perturbation (ASLP) which learns a unique label perturbation level for each training image. ASLP can significantly improve calibration degrees of dense binary classification models on both in-distribution and out-of-distribution data.
  arXiv  Detail & Related papers  (2023-07-25T14:40:11Z)
• Sample-dependent Adaptive Temperature Scaling for Improved Calibration [95.7477042886242]
  Post-hoc approach to compensate for neural networks being wrong is to perform temperature scaling. We propose to predict a different temperature value for each input, allowing us to adjust the mismatch between confidence and accuracy. We test our method on the ResNet50 and WideResNet28-10 architectures using the CIFAR10/100 and Tiny-ImageNet datasets.
  arXiv  Detail & Related papers  (2022-07-13T14:13:49Z)
• Localized Calibration: Metrics and Recalibration [133.07044916594361]
  We propose a fine-grained calibration metric that spans the gap between fully global and fully individualized calibration. We then introduce a localized recalibration method, LoRe, that improves the LCE better than existing recalibration methods.
  arXiv  Detail & Related papers  (2021-02-22T07:22:12Z)
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning [53.1047775185362]
  Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process.
  arXiv  Detail & Related papers  (2021-01-15T23:29:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.