SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)

• URL: http://arxiv.org/abs/2304.14803v1
• Date: Fri, 28 Apr 2023 12:20:35 GMT
• Title: SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)
• Authors: Elisa Leonardelli, Alexandra Uma, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Massimo Poesio
• Abstract summary: We report on the second edition of the LeWiDi series of shared tasks. This second edition attracted a wide array of participants resulting in 13 shared task submission papers.
• Score: 75.85548747729466
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: NLP datasets annotated with human judgments are rife with disagreements between the judges. This is especially true for tasks depending on subjective judgments such as sentiment analysis or offensive language detection. Particularly in these latter cases, the NLP community has come to realize that the approach of 'reconciling' these different subjective interpretations is inappropriate. Many NLP researchers have therefore concluded that rather than eliminating disagreements from annotated corpora, we should preserve them-indeed, some argue that corpora should aim to preserve all annotator judgments. But this approach to corpus creation for NLP has not yet been widely accepted. The objective of the LeWiDi series of shared tasks is to promote this approach to developing NLP models by providing a unified framework for training and evaluating with such datasets. We report on the second LeWiDi shared task, which differs from the first edition in three crucial respects: (i) it focuses entirely on NLP, instead of both NLP and computer vision tasks in its first edition; (ii) it focuses on subjective tasks, instead of covering different types of disagreements-as training with aggregated labels for subjective NLP tasks is a particularly obvious misrepresentation of the data; and (iii) for the evaluation, we concentrate on soft approaches to evaluation. This second edition of LeWiDi attracted a wide array of participants resulting in 13 shared task submission papers.

Related papers

• Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research [75.84463664853125]
  We provide a first attempt to quantify concerns regarding three topics, namely, environmental impact, equity, and impact on peer reviewing. We capture existing (dis)parities between different and within groups with respect to seniority, academia, and industry. We devise recommendations to mitigate found disparities, some of which already successfully implemented.
  arXiv  Detail & Related papers  (2023-06-29T12:44:53Z)
• No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference [6.485890157501745]
  Natural Language Inference (NLI) has been a cornerstone task in evaluating language models' inferential reasoning capabilities. Standard three-way classification scheme used in NLI has well-known shortcomings in evaluating models' ability to capture the nuances of natural human reasoning. We argue that the operationalization of the neutral label in current NLI datasets has low validity, is interpreted inconsistently, and that at least one important sense of neutrality is often ignored.
  arXiv  Detail & Related papers  (2023-06-16T15:45:08Z)
• We're Afraid Language Models Aren't Modeling Ambiguity [136.8068419824318]
  Managing ambiguity is a key part of human language understanding. We characterize ambiguity in a sentence by its effect on entailment relations with another sentence. We show that a multilabel NLI model can flag political claims in the wild that are misleading due to ambiguity.
  arXiv  Detail & Related papers  (2023-04-27T17:57:58Z)
• NLPeer: A Unified Resource for the Computational Study of Peer Review [58.71736531356398]
  We introduce NLPeer -- the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues. We augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information. Our work paves the path towards systematic, multi-faceted, evidence-based study of peer review in NLP and beyond.
  arXiv  Detail & Related papers  (2022-11-12T12:29:38Z)
• A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing [68.37496795076203]
  We provide guidance for NLP researchers and practitioners dealing with imbalanced data. We first discuss various types of controlled and real-world class imbalance. We organize the methods by whether they are based on sampling, data augmentation, choice of loss function, staged learning, or model design.
  arXiv  Detail & Related papers  (2022-10-10T13:26:40Z)
• Few-shot Named Entity Recognition with Cloze Questions [3.561183926088611]
  We propose a simple and intuitive adaptation of Pattern-Exploiting Training (PET), a recent approach which combines the cloze-questions mechanism and fine-tuning for few-shot learning. Our approach achieves considerably better performance than standard fine-tuning and comparable or improved results with respect to other few-shot baselines.
  arXiv  Detail & Related papers  (2021-11-24T11:08:59Z)
• ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning [97.10875695679499]
  We propose a novel contrastive learning framework named ERICA in pre-training phase to obtain a deeper understanding of the entities and their relations in text. Experimental results demonstrate that our proposed ERICA framework achieves consistent improvements on several document-level language understanding tasks.
  arXiv  Detail & Related papers  (2020-12-30T03:35:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.