On the Ramifications of Human Label Uncertainty

• URL: http://arxiv.org/abs/2211.05871v1
• Date: Thu, 10 Nov 2022 21:05:38 GMT
• Title: On the Ramifications of Human Label Uncertainty
• Abstract summary: We study the ramifications of human label uncertainty (HLU) We introduce a novel natural scene statistics (NSS) based label dilution training scheme without requiring massive human labels. Our experiments and analysis demonstrate that training with NSS-based label dilution alleviates the undue effects caused by HLU.
• Score: 17.906197972126325
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans exhibit disagreement during data labeling. We term this disagreement as human label uncertainty. In this work, we study the ramifications of human label uncertainty (HLU). Our evaluation of existing uncertainty estimation algorithms, with the presence of HLU, indicates the limitations of existing uncertainty metrics and algorithms themselves in response to HLU. Meanwhile, we observe undue effects in predictive uncertainty and generalizability. To mitigate the undue effects, we introduce a novel natural scene statistics (NSS) based label dilution training scheme without requiring massive human labels. Specifically, we first select a subset of samples with low perceptual quality ranked by statistical regularities of images. We then assign separate labels to each sample in this subset to obtain a training set with diluted labels. Our experiments and analysis demonstrate that training with NSS-based label dilution alleviates the undue effects caused by HLU.

Related papers

• An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration [2.8333175936728594]
  We present a controlled audit of soft-label learning across MNIST and a synthetic variant.<n>We show that while human soft-labels do provide accuracy gains, their larger value lies in acting as a regularizer.
  arXiv  Detail & Related papers  (2026-05-18T16:55:21Z)
• Detecting labeling bias using influence functions [1.9276091419970947]
  influence functions can be used to detect labeling bias.<n>We develop a sample valuation pipeline and test it first on the MNIST dataset.<n>On CheXpert, mislabeled samples consistently exhibit higher influence scores.
  arXiv  Detail & Related papers  (2026-02-22T11:20:35Z)
• Probabilistic Label Spreading: Efficient and Consistent Estimation of Soft Labels with Epistemic Uncertainty on Graphs [4.480864309234644]
  We introduce a probabilistic label spreading method that provides reliable estimates of aleatoric and epistemic uncertainty of labels.<n>We prove that label spreading yields consistent probability estimators even when the number of annotations per data point converges to zero.<n> Experimental results indicate that, compared to baselines, our approach substantially reduces the annotation budget required to achieve a desired label quality.
  arXiv  Detail & Related papers  (2026-02-04T14:00:30Z)
• Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge [51.93909886542317]
  We show how *relying on a single aggregate correlation score* can obscure fundamental differences between human labels and those from automatic evaluation. We propose stratifying data by human label uncertainty to provide a more robust analysis of automatic evaluation performance.
  arXiv  Detail & Related papers  (2024-10-03T03:08:29Z)
• From Biased Selective Labels to Pseudo-Labels: An Expectation-Maximization Framework for Learning from Biased Decisions [9.440055827786596]
  We study a clinically-inspired selective label problem called disparate censorship. Disparate Censorship Expectation-Maximization (DCEM) is an algorithm for learning in the presence of such censorship.
  arXiv  Detail & Related papers  (2024-06-27T03:33:38Z)
• Do not trust what you trust: Miscalibration in Semi-supervised Learning [21.20806568508201]
  State-of-the-art semi-supervised learning (SSL) approaches rely on highly confident predictions to serve as pseudo-labels that guide the training on unlabeled samples. We show that SSL methods based on pseudo-labels are significantly miscalibrated, and formally demonstrate the minimization of the min-entropy. We integrate a simple penalty term, which enforces the logit of the predictions on unlabeled samples to remain low, preventing the network predictions to become overconfident.
  arXiv  Detail & Related papers  (2024-03-22T18:43:46Z)
• Perceptual Quality-based Model Training under Annotator Label Uncertainty [15.015925663078377]
  Annotators exhibit disagreement during data labeling, which can be termed as annotator label uncertainty. We introduce a novel perceptual quality-based model training framework to objectively generate multiple labels for model training.
  arXiv  Detail & Related papers  (2024-03-15T10:52:18Z)
• Learning with Imbalanced Noisy Data by Preventing Bias in Sample Selection [82.43311784594384]
  Real-world datasets contain not only noisy labels but also class imbalance. We propose a simple yet effective method to address noisy labels in imbalanced datasets.
  arXiv  Detail & Related papers  (2024-02-17T10:34:53Z)
• Soft Curriculum for Learning Conditional GANs with Noisy-Labeled and Uncurated Unlabeled Data [70.25049762295193]
  We introduce a novel conditional image generation framework that accepts noisy-labeled and uncurated data during training. We propose soft curriculum learning, which assigns instance-wise weights for adversarial training while assigning new labels for unlabeled data. Our experiments show that our approach outperforms existing semi-supervised and label-noise robust methods in terms of both quantitative and qualitative performance.
  arXiv  Detail & Related papers  (2023-07-17T08:31:59Z)
• Complementing Semi-Supervised Learning with Uncertainty Quantification [6.612035830987296]
  We propose a novel unsupervised uncertainty-aware objective that relies on aleatoric and epistemic uncertainty quantification. Our results outperform the state-of-the-art results on complex datasets such as CIFAR-100 and Mini-ImageNet.
  arXiv  Detail & Related papers  (2022-07-22T00:15:02Z)
• Incorporating Label Uncertainty in Understanding Adversarial Robustness [17.65850501514483]
  We show that error regions induced by state-of-the-art models tend to have much higher label uncertainty compared with randomly-selected subsets. This observation motivates us to adapt a concentration estimation algorithm to account for label uncertainty.
  arXiv  Detail & Related papers  (2021-07-07T14:26:57Z)
• Sample Selection with Uncertainty of Losses for Learning with Noisy Labels [145.06552420999986]
  In learning with noisy labels, the sample selection approach is very popular, which regards small-loss data as correctly labeled during training. However, losses are generated on-the-fly based on the model being trained with noisy labels, and thus large-loss data are likely but not certainly to be incorrect. In this paper, we incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses.
  arXiv  Detail & Related papers  (2021-06-01T12:53:53Z)
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning [53.1047775185362]
  Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process.
  arXiv  Detail & Related papers  (2021-01-15T23:29:57Z)
• A Second-Order Approach to Learning with Instance-Dependent Label Noise [58.555527517928596]
  The presence of label noise often misleads the training of deep neural networks. We show that the errors in human-annotated labels are more likely to be dependent on the difficulty levels of tasks.
  arXiv  Detail & Related papers  (2020-12-22T06:36:58Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.