An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration

• URL: http://arxiv.org/abs/2605.18648v1
• Date: Mon, 18 May 2026 16:55:21 GMT
• Title: An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration
• Abstract summary: We present a controlled audit of soft-label learning across MNIST and a synthetic variant.<n>We show that while human soft-labels do provide accuracy gains, their larger value lies in acting as a regularizer.
• Score: 2.8333175936728594
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Central to human-aligned AI is understanding the benefits of human-elicited labels over synthetic alternatives. While human soft-labels improve calibration by capturing uncertainty, prior studies conflate these benefits with the implicit correction of mislabeled data (mode shifts), obscuring true effects of soft-labels. We present a controlled audit of soft-label learning across MNIST and a synthetic variant, re-annotating subsets to extract human uncertainty. By decoupling soft-label supervision from underlying label mode shifts, we show that while human soft-labels do provide accuracy gains, their larger value lies in acting as a regularizer that improves model calibration on difficult samples and promotes stable convergence across training runs. Dataset cartography reveals models trained on human soft-labels mirror human uncertainty, whereas those trained on synthetic labels fail to align with humans. Broadly, this work provides a diagnostic testbed for human-AI uncertainty alignment.

Related papers

• DiCaP: Distribution-Calibrated Pseudo-labeling for Semi-Supervised Multi-Label Learning [83.94574004953346]
  Semi-supervised multi-label learning aims to leverage unlabeled data to improve the model's performance.<n>Most existing methods assign equal weights to all pseudo-labels regardless of their quality.<n>We propose Distribution-Calibrated Pseudo-labeling (DiCaP), a correctness-aware framework that estimates posterior precision to calibrate pseudo-label weights.
  arXiv  Detail & Related papers  (2025-11-25T11:55:02Z)
• Soft-Label Training Preserves Epistemic Uncertainty [4.2541480167286005]
  Machine learning tasks involve inherent subjectivity, where annotators naturally provide varied labels.<n>Standard practice collapses these label distributions into single labels, aggregating diverse human judgments into point estimates.<n>We argue that this approach is misaligned for ambiguous data--the annotation distribution itself should be regarded as the ground truth.
  arXiv  Detail & Related papers  (2025-11-18T04:02:29Z)
• Reliable Wireless Indoor Localization via Cross-Validated Prediction-Powered Calibration [29.171927240331414]
  This letter proposes an approach that efficiently uses limited calibration data to simultaneously fine-tune a predictor and estimate the bias of synthetic labels.<n> Experiments on a fingerprinting dataset validate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2025-07-27T13:31:02Z)
• Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge [51.93909886542317]
  We show how *relying on a single aggregate correlation score* can obscure fundamental differences between human labels and those from automatic evaluation.<n>We propose stratifying data by human label uncertainty to provide a more robust analysis of automatic evaluation performance.
  arXiv  Detail & Related papers  (2024-10-03T03:08:29Z)
• Perceptual Quality-based Model Training under Annotator Label Uncertainty [15.015925663078377]
  Annotators exhibit disagreement during data labeling, which can be termed as annotator label uncertainty. We introduce a novel perceptual quality-based model training framework to objectively generate multiple labels for model training.
  arXiv  Detail & Related papers  (2024-03-15T10:52:18Z)
• Learning with Imbalanced Noisy Data by Preventing Bias in Sample Selection [82.43311784594384]
  Real-world datasets contain not only noisy labels but also class imbalance. We propose a simple yet effective method to address noisy labels in imbalanced datasets.
  arXiv  Detail & Related papers  (2024-02-17T10:34:53Z)
• SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning [101.86916775218403]
  This paper revisits the popular pseudo-labeling methods via a unified sample weighting formulation. We propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training. In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.
  arXiv  Detail & Related papers  (2023-01-26T03:53:25Z)
• On the Ramifications of Human Label Uncertainty [17.906197972126325]
  We study the ramifications of human label uncertainty (HLU) We introduce a novel natural scene statistics (NSS) based label dilution training scheme without requiring massive human labels. Our experiments and analysis demonstrate that training with NSS-based label dilution alleviates the undue effects caused by HLU.
  arXiv  Detail & Related papers  (2022-11-10T21:05:38Z)
• Unsupervised Domain Adaptive Salient Object Detection Through Uncertainty-Aware Pseudo-Label Learning [104.00026716576546]
  We propose to learn saliency from synthetic but clean labels, which naturally has higher pixel-labeling quality without the effort of manual annotations. We show that our proposed method outperforms the existing state-of-the-art deep unsupervised SOD methods on several benchmark datasets.
  arXiv  Detail & Related papers  (2022-02-26T16:03:55Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.