Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives!

• URL: http://arxiv.org/abs/2311.03153v1
• Date: Mon, 6 Nov 2023 14:47:48 GMT
• Title: Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives!
• Abstract summary: We consider the interplay of individual and shared perspectives in natural language processing. We investigate the extent to which architectures that include layers to model the relations between different annotators are beneficial for predicting single-annotator labels. Our findings indicate that approaches to subjectivity can benefit from relating individual perspectives.
• Score: 4.478789600295492
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many annotation tasks in natural language processing are highly subjective in that there can be different valid and justified perspectives on what is a proper label for a given example. This also applies to the judgment of argument quality, where the assignment of a single ground truth is often questionable. At the same time, there are generally accepted concepts behind argumentation that form a common ground. To best represent the interplay of individual and shared perspectives, we consider a continuum of approaches ranging from models that fully aggregate perspectives into a majority label to "share nothing"-architectures in which each annotator is considered in isolation from all other annotators. In between these extremes, inspired by models used in the field of recommender systems, we investigate the extent to which architectures that include layers to model the relations between different annotators are beneficial for predicting single-annotator labels. By means of two tasks of argument quality classification (argument concreteness and validity/novelty of conclusions), we show that recommender architectures increase the averaged annotator-individual F$_1$-scores up to $43\%$ over a majority label model. Our findings indicate that approaches to subjectivity can benefit from relating individual perspectives.

Related papers

• Learning Moral Diversity: Modelling Individual Perspectives in Moral Classification of Texts [7.927346467214517]
  We extend a pretrained language model with a layer that learns annotator-specific features.<n>Our model improves predictions of individual annotations and yields representations that reveal meaningful insights into annotators' moral perspectives.
  arXiv  Detail & Related papers  (2026-06-22T02:19:23Z)
• Coherence Maximization Improves Pluralistic Alignment [6.466328345684073]
  Internal Coherence Maximization (ICM) infers labels by maximizing their mutual predictability.<n>More coherent examples generalize substantially better than incoherent ones.<n>These results identify coherence as a key design principle for scalable value specification.
  arXiv  Detail & Related papers  (2026-06-02T03:56:03Z)
• QuMAB: Query-based Multi-Annotator Behavior Modeling with Reliability under Sparse Labels [23.555446749682467]
  Multi-annotator learning traditionally aggregates diverse annotations to approximate a single ground truth, treating disagreements as noise.<n>We introduce a paradigm shift from sample-wise aggregation to annotator-wise behavior modeling.<n>By treating annotator disagreements as valuable information rather than noise, modeling annotator-specific behavior patterns can reconstruct unlabeled data to reduce annotation cost, enhance aggregation reliability, and explain annotator decision behavior.
  arXiv  Detail & Related papers  (2025-07-23T16:17:43Z)
• Multi-Perspective Stance Detection [2.8073184910275293]
  Multi-perspective approach yields better classification performance than the baseline which uses the single label. This entails that designing more inclusive perspective-aware AI models is not only an essential first step in implementing responsible and ethical AI, but it can also achieve superior results than using the traditional approaches.
  arXiv  Detail & Related papers  (2024-11-13T16:30:41Z)
• Corpus Considerations for Annotator Modeling and Scaling [9.263562546969695]
  We show that the commonly used user token model consistently outperforms more complex models. Our findings shed light on the relationship between corpus statistics and annotator modeling performance.
  arXiv  Detail & Related papers  (2024-04-02T22:27:24Z)
• Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks [9.110872603799839]
  Supervised classification heavily depends on datasets annotated by humans. In subjective tasks such as toxicity classification, these annotations often exhibit low agreement among raters. In this work, we propose textbfAnnotator Awares for Texts (AART) for subjective classification tasks.
  arXiv  Detail & Related papers  (2023-11-16T10:18:32Z)
• Evaluating the Fairness of Discriminative Foundation Models in Computer Vision [51.176061115977774]
  We propose a novel taxonomy for bias evaluation of discriminative foundation models, such as Contrastive Language-Pretraining (CLIP) We then systematically evaluate existing methods for mitigating bias in these models with respect to our taxonomy. Specifically, we evaluate OpenAI's CLIP and OpenCLIP models for key applications, such as zero-shot classification, image retrieval and image captioning.
  arXiv  Detail & Related papers  (2023-10-18T10:32:39Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs) We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric. Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities [54.26896306906937]
  We present OVEN-Wiki, where a model need to link an image onto a Wikipedia entity with respect to a text query. We show that a PaLI-based auto-regressive visual recognition model performs surprisingly well, even on Wikipedia entities that have never been seen during fine-tuning. While PaLI-based models obtain higher overall performance, CLIP-based models are better at recognizing tail entities.
  arXiv  Detail & Related papers  (2023-02-22T05:31:26Z)
• SeedBERT: Recovering Annotator Rating Distributions from an Aggregated Label [43.23903984174963]
  We propose SeedBERT, a method for recovering annotator rating distributions from a single label. Our human evaluations indicate that SeedBERT's attention mechanism is consistent with human sources of annotator disagreement.
  arXiv  Detail & Related papers  (2022-11-23T18:35:15Z)
• Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations [6.546195629698355]
  We investigate the efficacy of multi-annotator models for subjective tasks. We show that this approach yields same or better performance than aggregating labels in the data prior to training. Our approach also provides a way to estimate uncertainty in predictions, which we demonstrate better correlate with annotation disagreements than traditional methods.
  arXiv  Detail & Related papers  (2021-10-12T03:12:34Z)
• Contrastive Explanations for Model Interpretability [77.92370750072831]
  We propose a methodology to produce contrastive explanations for classification models. Our method is based on projecting model representation to a latent space. Our findings shed light on the ability of label-contrastive explanations to provide a more accurate and finer-grained interpretability of a model's decision.
  arXiv  Detail & Related papers  (2021-03-02T00:36:45Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)
• Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations [67.4375210552593]
  We design experiments to understand an important but often ignored problem in visually grounded language generation. Given that humans have different utilities and visual attention, how will the sample variance in multi-reference datasets affect the models' performance? We show that it is of paramount importance to report variance in experiments; that human-generated references could vary drastically in different datasets/tasks, revealing the nature of each task.
  arXiv  Detail & Related papers  (2020-10-07T20:45:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.