AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling

• URL: http://arxiv.org/abs/2607.18465v1
• Date: Mon, 20 Jul 2026 19:32:01 GMT
• Title: AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling
• Abstract summary: AHEAD is a cross-annotator learning framework that advances annotator reliability estimation by leveraging the population-level data.<n>AHEAD substantially improves label accuracy, increasing average accuracy from 68.75% to 73.23%, with gains of up to 14.9% in the best case.
• Score: 7.8981698267367895
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Crowdsourced labeling provides valuable labeled data for domains across natural language processing, computer vision, and video. Label aggregation aims to infer latent true labels from noisy and biased annotations, with the key lying in annotator reliability estimation. Despite promising progress, existing approaches struggle with one real-world bottleneck: most individual annotators label only a small subset of tasks, making accurate annotator estimation highly intractable. In this paper, we focus on the considerably more challenging multi-class label aggregation and propose AHEAD (cross-Annotator learning and High-confidEnce Annotator-guideD label aggregation), a cross-annotator learning framework that advances annotator reliability estimation by leveraging the population-level data. Specifically, AHEAD first learns high-dimensional cross-annotator contexts via a graph neural network, deriving multi-view, complementary annotator embeddings by aggregating individual-level annotator features with contextual information. These embeddings are then decoded into interpretable annotator-specific confusion matrices to fit the observed labels. We formulate a composite objective incorporating high-confidence annotators to alleviate the unsupervised training issues faced by prior models. Experiments on 10 real-world datasets spanning NLP, CV, Video, and Audio show that AHEAD substantially improves label accuracy, increasing average accuracy from 68.75% to 73.23%, with gains of up to 14.9% in the best case. Meanwhile, scalability experiments on the largest dataset further demonstrate the overall superiority of our method.

Related papers

• Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition [55.189113121465816]
  We propose a novel correlation adaptation prompt network (CAPNET) for long-tailed multi-label visual recognition.<n>CAPNET explicitly models correlations from CLIP's textual encoder.<n>It improves generalization through test-time ensembling and realigns visual-textual modalities.
  arXiv  Detail & Related papers  (2025-11-25T18:57:28Z)
• Crowd-Certain: Label Aggregation in Crowdsourced and Ensemble Learning Classification [0.0]
  We introduce Crowd-Certain, a novel approach for label aggregation in crowdsourced and ensemble learning classification tasks. The proposed method uses the consistency of the annotators versus a trained classifier to determine a reliability score for each annotator. We extensively evaluated our approach against ten existing techniques across ten different datasets, each labeled by varying numbers of annotators.
  arXiv  Detail & Related papers  (2023-10-25T01:58:37Z)
• ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation [35.10805667891489]
  Active learning, as an annotation cost-saving strategy, has not been fully explored in the context of learning from disagreement. We show that in the active learning setting, a multi-head model performs significantly better than a single-head model in terms of uncertainty estimation.
  arXiv  Detail & Related papers  (2023-10-23T14:26:43Z)
• Probabilistic Test-Time Generalization by Variational Neighbor-Labeling [62.158807685159736]
  This paper strives for domain generalization, where models are trained exclusively on source domains before being deployed on unseen target domains. Probability pseudo-labeling of target samples to generalize the source-trained model to the target domain at test time. Variational neighbor labels that incorporate the information of neighboring target samples to generate more robust pseudo labels.
  arXiv  Detail & Related papers  (2023-07-08T18:58:08Z)
• Improving Classifier Robustness through Active Generation of Pairwise Counterfactuals [22.916599410472102]
  We present a novel framework that utilizes counterfactual generative models to generate a large number of diverse counterfactuals. We show that with a small amount of human-annotated counterfactual data (10%), we can generate a counterfactual augmentation dataset with learned labels.
  arXiv  Detail & Related papers  (2023-05-22T23:19:01Z)
• A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition [74.79785063365289]
  Existing models for named entity recognition (NER) are mainly based on large-scale labeled datasets. We propose a Confidence-based Partial Label Learning (CPLL) method to integrate the prior confidence (given by annotators) and posterior confidences (learned by models) for crowd-annotated NER.
  arXiv  Detail & Related papers  (2023-05-21T15:31:23Z)
• A Benchmark Generative Probabilistic Model for Weak Supervised Learning [2.0257616108612373]
  Weak Supervised Learning approaches have been developed to alleviate the annotation burden. We show that latent variable models (PLVMs) achieve state-of-the-art performance across four datasets.
  arXiv  Detail & Related papers  (2023-03-31T07:06:24Z)
• Debiased Pseudo Labeling in Self-Training [77.83549261035277]
  Deep neural networks achieve remarkable performances on a wide range of tasks with the aid of large-scale labeled datasets. To mitigate the requirement for labeled data, self-training is widely used in both academia and industry by pseudo labeling on readily-available unlabeled data. We propose Debiased, in which the generation and utilization of pseudo labels are decoupled by two independent heads.
  arXiv  Detail & Related papers  (2022-02-15T02:14:33Z)
• Boosting Semi-Supervised Face Recognition with Noise Robustness [54.342992887966616]
  This paper presents an effective solution to semi-supervised face recognition that is robust to the label noise aroused by the auto-labelling. We develop a semi-supervised face recognition solution, named Noise Robust Learning-Labelling (NRoLL), which is based on the robust training ability empowered by GN.
  arXiv  Detail & Related papers  (2021-05-10T14:43:11Z)
• Bayesian Semi-supervised Crowdsourcing [71.20185379303479]
  Crowdsourcing has emerged as a powerful paradigm for efficiently labeling large datasets and performing various learning tasks. This work deals with semi-supervised crowdsourced classification, under two regimes of semi-supervision.
  arXiv  Detail & Related papers  (2020-12-20T23:18:51Z)
• End-to-End Learning from Noisy Crowd to Supervised Machine Learning Models [6.278267504352446]
  We advocate using hybrid intelligence, i.e., combining deep models and human experts, to design an end-to-end learning framework from noisy crowd-sourced data. We show how label aggregation can benefit from estimating the annotators' confusion matrix to improve the learning process. We demonstrate the effectiveness of our strategies on several image datasets, using SVM and deep neural networks.
  arXiv  Detail & Related papers  (2020-11-13T09:48:30Z)
• Adversarial Knowledge Transfer from Unlabeled Data [62.97253639100014]
  We present a novel Adversarial Knowledge Transfer framework for transferring knowledge from internet-scale unlabeled data to improve the performance of a classifier. An important novel aspect of our method is that the unlabeled source data can be of different classes from those of the labeled target data, and there is no need to define a separate pretext task.
  arXiv  Detail & Related papers  (2020-08-13T08:04:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.