Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks

• URL: http://arxiv.org/abs/2605.09955v1
• Date: Mon, 11 May 2026 04:04:03 GMT
• Title: Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks
• Abstract summary: Disagreement in annotation is a common phenomenon in the development of NLP datasets.<n>We propose an agreement-based clustering technique to model the disagreement between the annotators.
• Score: 39.645768996277184
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Disagreement in annotation is a common phenomenon in the development of NLP datasets and serves as a valuable source of insight. While majority voting remains the dominant strategy for aggregating labels, recent work has explored modeling individual annotators to preserve their perspectives. However, modeling each annotator is resource-intensive and remains underexplored across various NLP tasks. We propose an agreement-based clustering technique to model the disagreement between the annotators. We conduct comprehensive experiments in 40 datasets in 18 typologically diverse languages, covering three subjective NLP tasks: sentiment analysis, emotion classification, and hate speech detection. We evaluate four aggregation approaches: majority vote, ensemble, multi-label, and multitask. The results demonstrate that agreement-based clustering can leverage the full spectrum of annotator perspectives and significantly enhance classification performance in subjective NLP tasks compared to majority voting and individual annotator modeling. Regarding the aggregation approach, the multi-label and multitask approaches are better for modeling clustered annotators than an ensemble and model majority vote.

Related papers

• Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition [86.4487145812318]
  Fine-grained action recognition in egocentric video is challenging for Vision-Language Models.<n>We propose Divide, Deliberate, Decide, a fully-local, zero-shot multi-agent framework.<n> Experiments show that our method positively improves zero-shot action recognition performance over the baseline.
  arXiv  Detail & Related papers  (2026-06-16T07:31:27Z)
• Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models [60.9932576020062]
  Test-time compute strategies have emerged as a lightweight approach to boost reasoning in large language models.<n>We propose Entropy-based TTC, which selects the most confident prediction based on predictive entropy.<n>We prove that ETTC outperforms majority voting under mild assumptions and empirically demonstrate that it consistently surpasses both voting and the best individual model.
  arXiv  Detail & Related papers  (2026-05-29T01:06:38Z)
• The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants [66.6636608563034]
  We present the Avengers -- a simple recipe that leverages the collective intelligence of smaller models.<n>With 10 open-source models, the Avengers surpasses GPT-4o, 4.1, and 4.5 in average performance across 15 diverse datasets.<n>In particular, it surpasses GPT-4.1 on mathematics tasks by 18.21% and on code tasks by 7.46%.
  arXiv  Detail & Related papers  (2025-05-26T10:29:42Z)
• Voices in a Crowd: Searching for Clusters of Unique Perspectives [8.516397617576978]
  Proposed solutions aim to capture minority perspectives by either modelling annotator disagreements or grouping annotators based on shared metadata. We propose a framework that trains models without encoding annotator metadata, extracts latent embeddings informed by annotator behaviour, and creates clusters of similar opinions.
  arXiv  Detail & Related papers  (2024-07-19T12:37:15Z)
• Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances [24.142013877384603]
  This paper introduces a novel unsupervised multimodal clustering method (UMC), making a pioneering contribution to this field. UMC introduces a unique approach to constructing augmentation views for multimodal data, which are then used to perform pre-training. We show remarkable improvements of 2-6% scores in clustering metrics over state-of-the-art methods, marking the first successful endeavor in this domain.
  arXiv  Detail & Related papers  (2024-05-21T13:24:07Z)
• Annotator-Centric Active Learning for Subjective NLP Tasks [7.766754308448708]
  Active Learning (AL) addresses the high costs of collecting human annotations by strategically annotating the most informative samples. We introduce Annotator-Centric Active Learning (ACAL), which incorporates an annotator selection strategy following data sampling. Our objective is to efficiently approximate the full diversity of human judgments, and to assess model performance using annotator-centric metrics.
  arXiv  Detail & Related papers  (2024-04-24T08:13:02Z)
• Corpus Considerations for Annotator Modeling and Scaling [9.263562546969695]
  We show that the commonly used user token model consistently outperforms more complex models. Our findings shed light on the relationship between corpus statistics and annotator modeling performance.
  arXiv  Detail & Related papers  (2024-04-02T22:27:24Z)
• MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models [70.92847554971065]
  We introduce MT-Eval, a comprehensive benchmark designed to evaluate multi-turn conversational abilities. By analyzing human-LLM conversations, we categorize interaction patterns into four types: recollection, expansion, refinement, and follow-up. Our evaluation of 11 well-known LLMs shows that while closed-source models generally surpass open-source ones, certain open-source models exceed GPT-3.5-Turbo in specific tasks.
  arXiv  Detail & Related papers  (2024-01-30T04:50:28Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• Unified Multi-View Orthonormal Non-Negative Graph Based Clustering Framework [74.25493157757943]
  We formulate a novel clustering model, which exploits the non-negative feature property and incorporates the multi-view information into a unified joint learning framework. We also explore, for the first time, the multi-model non-negative graph-based approach to clustering data based on deep features.
  arXiv  Detail & Related papers  (2022-11-03T08:18:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.