Quantifying Ambiguity in Categorical Annotations: A Measure and Statistical Inference Framework

• URL: http://arxiv.org/abs/2510.04366v1
• Date: Sun, 05 Oct 2025 21:19:42 GMT
• Title: Quantifying Ambiguity in Categorical Annotations: A Measure and Statistical Inference Framework
• Authors: Christopher Klugmann, Daniel Kondermann,
• Abstract summary: We introduce an ambiguity measure that maps a discrete response distribution to a scalar in the unit interval.<n>We analyze the measure's formal properties and contrast its behavior with a representative ambiguity measure from the literature.
• Score: 0.7180881562002392
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human-generated categorical annotations frequently produce empirical response distributions (soft labels) that reflect ambiguity rather than simple annotator error. We introduce an ambiguity measure that maps a discrete response distribution to a scalar in the unit interval, designed to quantify aleatoric uncertainty in categorical tasks. The measure bears a close relationship to quadratic entropy (Gini-style impurity) but departs from those indices by treating an explicit "can't solve" category asymmetrically, thereby separating uncertainty arising from class-level indistinguishability from uncertainty due to explicit unresolvability. We analyze the measure's formal properties and contrast its behavior with a representative ambiguity measure from the literature. Moving beyond description, we develop statistical tools for inference: we propose frequentist point estimators for population ambiguity and derive the Bayesian posterior over ambiguity induced by Dirichlet priors on the underlying probability vector, providing a principled account of epistemic uncertainty. Numerical examples illustrate estimation, calibration, and practical use for dataset-quality assessment and downstream machine-learning workflows.

Related papers

• Credal Concept Bottleneck Models: Structural Separation of Epistemic and Aleatoric Uncertainty [38.040646841317965]
  We propose a credal-set formulation in which uncertainty is represented as a set of predictive distributions.<n>We instantiate this idea in a Variational Credal Concept Bottleneck Model with two disjoint uncertainty heads trained by disjoint objectives.
  arXiv  Detail & Related papers  (2026-02-11T10:54:57Z)
• The Illusion of Certainty: Uncertainty quantification for LLMs fails under ambiguity [48.899855816199484]
  We introduce MAQA* and AmbigQA*, the first ambiguous question-answering (QA) datasets equipped with ground-truth answer distributions.<n>We show that predictive-distribution and ensemble-based estimators are fundamentally limited under ambiguity.
  arXiv  Detail & Related papers  (2025-11-06T14:46:35Z)
• Uncertainty Estimation using Variance-Gated Distributions [0.6340400318304492]
  We propose an intuitive framework for uncertainty estimation and decomposition based on the signal-to-noise ratio of class probability distributions.<n>We introduce a variance-gated measure that scales predictions by a confidence factor derived from ensembles.
  arXiv  Detail & Related papers  (2025-09-07T16:19:21Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Quantification of Predictive Uncertainty via Inference-Time Sampling [57.749601811982096]
  We propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions.
  arXiv  Detail & Related papers  (2023-08-03T12:43:21Z)
• Understanding Uncertainty Sampling [7.32527270949303]
  Uncertainty sampling is a prevalent active learning algorithm that queries sequentially the annotations of data samples. We propose a notion of equivalent loss which depends on the used uncertainty measure and the original loss function. We provide the first generalization bound for uncertainty sampling algorithms under both stream-based and pool-based settings.
  arXiv  Detail & Related papers  (2023-07-06T01:57:37Z)
• Doubly Robust Counterfactual Classification [1.8907108368038217]
  We study counterfactual classification as a new tool for decision-making under hypothetical (contrary to fact) scenarios. We propose a doubly-robust nonparametric estimator for a general counterfactual classifier.
  arXiv  Detail & Related papers  (2023-01-15T22:04:46Z)
• The Implicit Delta Method [61.36121543728134]
  In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of uncertainty. We show that the change in the evaluation due to regularization is consistent for the variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference.
  arXiv  Detail & Related papers  (2022-11-11T19:34:17Z)
• Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition [59.52434325897716]
  We propose a solution, named DMUE, to address the problem of annotation ambiguity from two perspectives. For the former, an auxiliary multi-branch learning framework is introduced to better mine and describe the latent distribution in the label space. For the latter, the pairwise relationship of semantic feature between instances are fully exploited to estimate the ambiguity extent in the instance space.
  arXiv  Detail & Related papers  (2021-04-01T03:21:57Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)
• Nonparametric Score Estimators [49.42469547970041]
  Estimating the score from a set of samples generated by an unknown distribution is a fundamental task in inference and learning of probabilistic models. We provide a unifying view of these estimators under the framework of regularized nonparametric regression. We propose score estimators based on iterative regularization that enjoy computational benefits from curl-free kernels and fast convergence.
  arXiv  Detail & Related papers  (2020-05-20T15:01:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.