End-to-End Annotator Bias Approximation on Crowdsourced Single-Label Sentiment Analysis

• URL: http://arxiv.org/abs/2111.02326v2
• Date: Mon, 24 Jul 2023 19:44:53 GMT
• Title: End-to-End Annotator Bias Approximation on Crowdsourced Single-Label Sentiment Analysis
• Authors: Gerhard Johann Hagerer, David Szabo, Andreas Koch, Maria Luisa Ripoll Dominguez, Christian Widmer, Maximilian Wich, Hannah Danner, Georg Groh
• Abstract summary: Sentiment analysis is often a crowdsourcing task prone to subjective labels given by many annotators. It is not yet fully understood how the annotation bias of each annotator can be modeled correctly with state-of-the-art methods. Our contribution is an explanation and improvement for precise neural end-to-end bias modeling and ground truth estimation.
• Score: 0.4925222726301579
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sentiment analysis is often a crowdsourcing task prone to subjective labels given by many annotators. It is not yet fully understood how the annotation bias of each annotator can be modeled correctly with state-of-the-art methods. However, resolving annotator bias precisely and reliably is the key to understand annotators' labeling behavior and to successfully resolve corresponding individual misconceptions and wrongdoings regarding the annotation task. Our contribution is an explanation and improvement for precise neural end-to-end bias modeling and ground truth estimation, which reduces an undesired mismatch in that regard of the existing state-of-the-art. Classification experiments show that it has potential to improve accuracy in cases where each sample is annotated only by one single annotator. We provide the whole source code publicly and release an own domain-specific sentiment dataset containing 10,000 sentences discussing organic food products. These are crawled from social media and are singly labeled by 10 non-expert annotators.

Related papers

• Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains [60.5207173547769]
  We evaluate zero-shot generated summaries across specialized domains including biomedical articles, and legal bills. We acquire annotations from domain experts to identify inconsistencies in summaries and systematically categorize these errors. We release all collected annotations to facilitate additional research toward measuring and realizing factually accurate summarization, beyond news articles.
  arXiv  Detail & Related papers  (2024-02-05T20:51:11Z)
• Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks [9.110872603799839]
  Supervised classification heavily depends on datasets annotated by humans. In subjective tasks such as toxicity classification, these annotations often exhibit low agreement among raters. In this work, we propose textbfAnnotator Awares for Texts (AART) for subjective classification tasks.
  arXiv  Detail & Related papers  (2023-11-16T10:18:32Z)
• Probabilistic Test-Time Generalization by Variational Neighbor-Labeling [62.158807685159736]
  This paper strives for domain generalization, where models are trained exclusively on source domains before being deployed on unseen target domains. Probability pseudo-labeling of target samples to generalize the source-trained model to the target domain at test time. Variational neighbor labels that incorporate the information of neighboring target samples to generate more robust pseudo labels.
  arXiv  Detail & Related papers  (2023-07-08T18:58:08Z)
• SeedBERT: Recovering Annotator Rating Distributions from an Aggregated Label [43.23903984174963]
  We propose SeedBERT, a method for recovering annotator rating distributions from a single label. Our human evaluations indicate that SeedBERT's attention mechanism is consistent with human sources of annotator disagreement.
  arXiv  Detail & Related papers  (2022-11-23T18:35:15Z)
• Utilizing supervised models to infer consensus labels and their quality from data with multiple annotators [16.79939549201032]
  Real-world data for classification is often labeled by multiple annotators. We introduce CROWDLAB, a straightforward approach to estimate such data. Our proposed method provides superior estimates for (1)- (3) than many alternative algorithms.
  arXiv  Detail & Related papers  (2022-10-13T07:54:07Z)
• Self-supervised debiasing using low rank regularization [59.84695042540525]
  Spurious correlations can cause strong biases in deep neural networks, impairing generalization ability. We propose a self-supervised debiasing framework potentially compatible with unlabeled samples. Remarkably, the proposed debiasing framework significantly improves the generalization performance of self-supervised learning baselines.
  arXiv  Detail & Related papers  (2022-10-11T08:26:19Z)
• Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations [6.546195629698355]
  We investigate the efficacy of multi-annotator models for subjective tasks. We show that this approach yields same or better performance than aggregating labels in the data prior to training. Our approach also provides a way to estimate uncertainty in predictions, which we demonstrate better correlate with annotation disagreements than traditional methods.
  arXiv  Detail & Related papers  (2021-10-12T03:12:34Z)
• Learning from Crowds with Sparse and Imbalanced Annotations [29.596070201105274]
  crowdsourcing has established itself as an efficient labeling solution through resorting to non-expert crowds. One common practice is to distribute each instance to multiple workers, whereas each worker only annotates a subset of data, resulting in the it sparse annotation phenomenon. We propose one self-training based approach named it Self-Crowd by progressively adding confident pseudo-annotations and rebalancing the annotation distribution.
  arXiv  Detail & Related papers  (2021-07-11T13:06:20Z)
• Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces [64.23172847182109]
  We show that different negative sampling schemes implicitly trade-off performance on dominant versus rare labels. We provide a unified means to explicitly tackle both sampling bias, arising from working with a subset of all labels, and labeling bias, which is inherent to the data due to label imbalance.
  arXiv  Detail & Related papers  (2021-05-12T15:40:13Z)
• Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets [90.61266099147053]
  We investigate efficient annotation strategies for collecting multi-class classification labels for a large collection of images. We propose modifications and best practices aimed at minimizing human labeling effort. Simulated experiments on a 125k image subset of the ImageNet100 show that it can be annotated to 80% top-1 accuracy with 0.35 annotations per image on average.
  arXiv  Detail & Related papers  (2021-04-26T16:29:32Z)
• One-bit Supervision for Image Classification [121.87598671087494]
  One-bit supervision is a novel setting of learning from incomplete annotations. We propose a multi-stage training paradigm which incorporates negative label suppression into an off-the-shelf semi-supervised learning algorithm.
  arXiv  Detail & Related papers  (2020-09-14T03:06:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.