Revisiting Active Learning under (Human) Label Variation

• URL: http://arxiv.org/abs/2507.02593v1
• Date: Thu, 03 Jul 2025 12:59:28 GMT
• Title: Revisiting Active Learning under (Human) Label Variation
• Abstract summary: Human label variation (HLV) is the occurrence of plausible differences in annotations as an informative signal.<n>We propose a conceptual framework for incorporating HLV throughout the active learning loop, including instance selection, annotator choice, and label representation.<n>Our work aims to lay a conceptual foundation for HLV-aware active learning, better reflecting the complexities of real-world annotation.
• Score: 25.25602927653135
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Access to high-quality labeled data remains a limiting factor in applied supervised learning. While label variation (LV), i.e., differing labels for the same instance, is common, especially in natural language processing, annotation frameworks often still rest on the assumption of a single ground truth. This overlooks human label variation (HLV), the occurrence of plausible differences in annotations, as an informative signal. Similarly, active learning (AL), a popular approach to optimizing the use of limited annotation budgets in training ML models, often relies on at least one of several simplifying assumptions, which rarely hold in practice when acknowledging HLV. In this paper, we examine foundational assumptions about truth and label nature, highlighting the need to decompose observed LV into signal (e.g., HLV) and noise (e.g., annotation error). We survey how the AL and (H)LV communities have addressed -- or neglected -- these distinctions and propose a conceptual framework for incorporating HLV throughout the AL loop, including instance selection, annotator choice, and label representation. We further discuss the integration of large language models (LLM) as annotators. Our work aims to lay a conceptual foundation for HLV-aware active learning, better reflecting the complexities of real-world annotation.

Related papers

• Leveraging Vision-Language Models as Weak Annotators in Active Learning [13.163098563588727]
  Vision-language models (VLMs) can be leveraged to reduce the reliance on costly human annotation.<n>We propose an active learning framework that combines fine-grained human annotations with coarse-grained VLM-generated weak labels.
  arXiv  Detail & Related papers  (2026-05-01T07:40:49Z)
• From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLP [35.8165102355067]
  We argue that preserving HLV as an embodiment of human pluralism must be treated as a Selbstzweck.<n>We call for proactively incorporating HLV into preference datasets and outline actionable steps towards it.
  arXiv  Detail & Related papers  (2025-10-09T17:48:29Z)
• Can Large Language Models Capture Human Annotator Disagreements? [84.32752330104775]
  Large Language Models (LLMs) are increasingly used for automatic annotation to reduce human effort.<n>It is still unclear, however, whether these models also capture informative human annotation variation.<n>Our work addresses this gap by extensively evaluating LLMs' ability to predict annotation disagreements without access to repeated human labels.
  arXiv  Detail & Related papers  (2025-06-24T09:49:26Z)
• Bridging Weakly-Supervised Learning and VLM Distillation: Noisy Partial Label Learning for Efficient Downstream Adaptation [51.67328507400985]
  In noisy partial label learning (NPLL), each training sample is associated with a set of candidate labels annotated by multiple noisy annotators.<n>This paper focuses on learning from partial labels annotated by pre-trained vision-language models.<n>It proposes an innovative collaborative consistency regularization (Co-Reg) method.
  arXiv  Detail & Related papers  (2025-06-03T12:48:54Z)
• Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts [26.415262737856967]
  We explore label verification in contexts using Large Language Models (LLMs)<n>We propose the Label-in-a-Haystack Rectification (LiaHR) framework for subjective label correction.<n>This approach can be integrated into annotation pipelines to enhance signal-to-noise ratios.
  arXiv  Detail & Related papers  (2025-05-22T18:55:22Z)
• Semantic-guided Representation Learning for Multi-Label Recognition [13.046479112800608]
  Multi-label Recognition (MLR) involves assigning multiple labels to each data instance in an image.<n>Recent Vision and Language Pre-training methods have made significant progress in tackling zero-shot MLR tasks.<n>We introduce a Semantic-guided Representation Learning approach (SigRL) that enables the model to learn effective visual and textual representations.
  arXiv  Detail & Related papers  (2025-04-04T08:15:08Z)
• Incremental Label Distribution Learning with Scalable Graph Convolutional Networks [41.02170058889797]
  We introduce Incremental Label Distribution Learning (ILDL), analyze its key issues regarding training samples and inter-label relationships. Specifically, we develop a New-label-aware Gradient Compensation Loss to speed up the learning of new labels and represent inter-label relationships as a graph.
  arXiv  Detail & Related papers  (2024-11-20T07:49:51Z)
• Exploring Vacant Classes in Label-Skewed Federated Learning [113.65301899666645]
  This paper introduces FedVLS, a novel approach to label-skewed federated learning.<n>It integrates vacant-class distillation and logit suppression simultaneously.<n>Experiments validate the efficacy of FedVLS, demonstrating superior performance compared to previous state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2024-01-04T16:06:31Z)
• Virtual Category Learning: A Semi-Supervised Learning Method for Dense Prediction with Extremely Limited Labels [63.16824565919966]
  This paper proposes to use confusing samples proactively without label correction. A Virtual Category (VC) is assigned to each confusing sample in such a way that it can safely contribute to the model optimisation. Our intriguing findings highlight the usage of VC learning in dense vision tasks.
  arXiv  Detail & Related papers  (2023-12-02T16:23:52Z)
• Robust Representation Learning for Unreliable Partial Label Learning [86.909511808373]
  Partial Label Learning (PLL) is a type of weakly supervised learning where each training instance is assigned a set of candidate labels, but only one label is the ground-truth. This is known as Unreliable Partial Label Learning (UPLL) that introduces an additional complexity due to the inherent unreliability and ambiguity of partial labels. We propose the Unreliability-Robust Representation Learning framework (URRL) that leverages unreliability-robust contrastive learning to help the model fortify against unreliable partial labels effectively.
  arXiv  Detail & Related papers  (2023-08-31T13:37:28Z)
• Channel-Wise Contrastive Learning for Learning with Noisy Labels [60.46434734808148]
  We introduce channel-wise contrastive learning (CWCL) to distinguish authentic label information from noise. Unlike conventional instance-wise contrastive learning (IWCL), CWCL tends to yield more nuanced and resilient features aligned with the authentic labels. Our strategy is twofold: firstly, using CWCL to extract pertinent features to identify cleanly labeled samples, and secondly, progressively fine-tuning using these samples.
  arXiv  Detail & Related papers  (2023-08-14T06:04:50Z)
• Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need [34.013568381942775]
  Self-Supervised Learning (SSL) has emerged as the solution of choice to learn transferable representations from unlabeled data. In this work, we formalize and generalize this principle through Positive Active Learning (PAL) where an oracle queries semantic relationships between samples. First, it unveils a theoretically grounded learning framework beyond SSL, based on similarity graphs, that can be extended to tackle supervised and semi-supervised learning depending on the employed oracle. Second, it provides a consistent algorithm to embed a priori knowledge, e.g. some observed labels, into any SSL losses without any change in the training pipeline.
  arXiv  Detail & Related papers  (2023-03-27T14:44:39Z)
• A Deep Model for Partial Multi-Label Image Classification with Curriculum Based Disambiguation [42.0958430465578]
  We study the partial multi-label (PML) image classification problem. Existing PML methods typically design a disambiguation strategy to filter out noisy labels. We propose a deep model for PML to enhance the representation and discrimination ability.
  arXiv  Detail & Related papers  (2022-07-06T02:49:02Z)
• Active Refinement for Multi-Label Learning: A Pseudo-Label Approach [84.52793080276048]
  Multi-label learning (MLL) aims to associate a given instance with its relevant labels from a set of concepts. Previous works of MLL mainly focused on the setting where the concept set is assumed to be fixed. Many real-world applications require introducing new concepts into the set to meet new demands.
  arXiv  Detail & Related papers  (2021-09-29T19:17:05Z)
• The Role of Global Labels in Few-Shot Classification and How to Infer Them [55.64429518100676]
  Few-shot learning is a central problem in meta-learning, where learners must quickly adapt to new tasks. We propose Meta Label Learning (MeLa), a novel algorithm that infers global labels and obtains robust few-shot models via standard classification.
  arXiv  Detail & Related papers  (2021-08-09T14:07:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.