From Ground Truth to Measurement: A Statistical Framework for Human Labeling

• URL: http://arxiv.org/abs/2604.07591v1
• Date: Wed, 08 Apr 2026 20:49:03 GMT
• Title: From Ground Truth to Measurement: A Statistical Framework for Human Labeling
• Abstract summary: Supervised machine learning assumes that labeled data provide accurate measurements of the concepts models are meant to learn.<n>Yet in practice, human labeling introduces systematic variation arising from ambiguous items, divergent interpretations, and simple mistakes.<n>This paper reframes annotation as a measurement process and introduces a statistical framework for decomposing labeling outcomes into interpretable sources of variation.
• Score: 6.5507170110120185
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Supervised machine learning assumes that labeled data provide accurate measurements of the concepts models are meant to learn. Yet in practice, human labeling introduces systematic variation arising from ambiguous items, divergent interpretations, and simple mistakes. Machine learning research commonly treats all disagreement as noise, which obscures these distinctions and limits our understanding of what models actually learn. This paper reframes annotation as a measurement process and introduces a statistical framework for decomposing labeling outcomes into interpretable sources of variation: instance difficulty, annotator bias, situational noise, and relational alignment. The framework extends classical measurement-error models to accommodate both shared and individualized notions of truth, reflecting traditional and human label variation interpretations of error, and provides a diagnostic for assessing which regime better characterizes a given task. Applying the proposed model to a multi-annotator natural language inference dataset, we find empirical evidence for all four theorized components and demonstrate the effectiveness of our approach. We conclude with implications for data-centric machine learning and outline how this approach can guide the development of a more systematic science of labeling.

Related papers

• Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration [0.21410799064827235]
  Large language models (LLMs) are increasingly positioned as scalable tools for annotating educational data.<n>However, growing evidence suggests that single-pass LLM outputs remain unreliable for high-stakes educational constructs.<n>This tension between scale and validity sits at the core of contemporary education data science.
  arXiv  Detail & Related papers  (2026-03-08T16:51:03Z)
• Tutorial on the Probabilistic Unification of Estimation Theory, Machine Learning, and Generative AI [0.0]
  This survey presents a unified mathematical framework that connects classical estimation theory, statistical inference, and modern machine learning.<n>We show how techniques such as maximum likelihood estimation, Bayesian inference, and attention mechanisms address uncertainty.<n>It serves as both a theoretical synthesis and a practical guide for students and researchers navigating the evolving landscape of machine learning.
  arXiv  Detail & Related papers  (2025-08-21T16:57:33Z)
• Convergence Behavior of an Adversarial Weak Supervision Method [10.409652277630133]
  Weak Supervision is a paradigm subsuming subareas of machine learning. By using labeled data to train modern machine learning methods, the cost of acquiring large amounts of hand labeled data can be ameliorated. Two approaches to combining the rules-of-thumb falls into two camps, reflecting different ideologies of statistical estimation.
  arXiv  Detail & Related papers  (2024-05-25T02:33:17Z)
• Differential contributions of machine learning and statistical analysis to language and cognitive sciences [27.152245569974678]
  This study employs the Buckeye Speech Corpus to illustrate how machine learning and statistical analysis are applied in data-driven research. We demonstrate the theoretical differences, implementation steps, and unique objectives of each approach. The study highlights how semantic relevance, a novel metric measuring contextual influence on target words, contributes to understanding word duration in speech.
  arXiv  Detail & Related papers  (2024-04-22T10:06:21Z)
• Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective [68.20531518525273]
  We take a closer look into existing self-supervised methods of speech from an information-theoretic perspective. We use linear probes to estimate the mutual information between the target information and learned representations. We explore the potential of evaluating representations in a self-supervised fashion, where we estimate the mutual information between different parts of the data without using any labels.
  arXiv  Detail & Related papers  (2024-01-16T21:13:22Z)
• Bias-inducing geometries: an exactly solvable data model with fairness implications [12.532003449620607]
  We introduce an exactly solvable high-dimensional model of data imbalance.<n>We analytically unpack the typical properties of learning models trained in this synthetic framework.<n>We obtain exact predictions for the observables that are commonly employed for fairness assessment.
  arXiv  Detail & Related papers  (2022-05-31T16:27:57Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• Resolving label uncertainty with implicit posterior models [71.62113762278963]
  We propose a method for jointly inferring labels across a collection of data samples. By implicitly assuming the existence of a generative model for which a differentiable predictor is the posterior, we derive a training objective that allows learning under weak beliefs.
  arXiv  Detail & Related papers  (2022-02-28T18:09:44Z)
• Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations [64.85696493596821]
  In computer vision applications, generative counterfactual methods indicate how to perturb a model's input to change its prediction. We propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss. Our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-18T12:57:34Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)
• Fairness in Semi-supervised Learning: Unlabeled Data Help to Reduce Discrimination [53.3082498402884]
  A growing specter in the rise of machine learning is whether the decisions made by machine learning models are fair. We present a framework of fair semi-supervised learning in the pre-processing phase, including pseudo labeling to predict labels for unlabeled data. A theoretical decomposition analysis of bias, variance and noise highlights the different sources of discrimination and the impact they have on fairness in semi-supervised learning.
  arXiv  Detail & Related papers  (2020-09-25T05:48:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.