Investigating Reasons for Disagreement in Natural Language Inference
- URL: http://arxiv.org/abs/2209.03392v1
- Date: Wed, 7 Sep 2022 18:01:39 GMT
- Title: Investigating Reasons for Disagreement in Natural Language Inference
- Authors: Nan-Jiang Jiang and Marie-Catherine de Marneffe
- Abstract summary: We develop a taxonomy of disagreement sources spanning 3 high-level classes.
Some disagreements are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts.
We explore two modeling approaches for detecting items with potential disagreement.
- Score: 9.002603216969154
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We investigate how disagreement in natural language inference (NLI)
annotation arises. We developed a taxonomy of disagreement sources with 10
categories spanning 3 high-level classes. We found that some disagreements are
due to uncertainty in the sentence meaning, others to annotator biases and task
artifacts, leading to different interpretations of the label distribution. We
explore two modeling approaches for detecting items with potential
disagreement: a 4-way classification with a "Complicated" label in addition to
the three standard NLI labels, and a multilabel classification approach. We
found that the multilabel classification is more expressive and gives better
recall of the possible interpretations in the data.
Related papers
- Online Multi-Label Classification under Noisy and Changing Label Distribution [9.17381554071824]
We propose an online multi-label classification algorithm under Noisy and Changing Label Distribution (NCLD)
The objective is to simultaneously model the label scoring and the label ranking for high accuracy, whose robustness to NCLD benefits from three novel works.
arXiv Detail & Related papers (2024-10-03T11:16:43Z) - Leveraging Annotator Disagreement for Text Classification [3.6625157427847963]
It is common practice in text classification to only use one majority label for model training even if a dataset has been annotated by multiple annotators.
This paper proposes three strategies to leverage annotator disagreement for text classification: a probability-based multi-label method, an ensemble system, and instruction tuning.
arXiv Detail & Related papers (2024-09-26T06:46:53Z) - RankCSE: Unsupervised Sentence Representations Learning via Learning to
Rank [54.854714257687334]
We propose a novel approach, RankCSE, for unsupervised sentence representation learning.
It incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework.
An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks.
arXiv Detail & Related papers (2023-05-26T08:27:07Z) - Towards Unsupervised Recognition of Token-level Semantic Differences in
Related Documents [61.63208012250885]
We formulate recognizing semantic differences as a token-level regression task.
We study three unsupervised approaches that rely on a masked language model.
Our results show that an approach based on word alignment and sentence-level contrastive learning has a robust correlation to gold labels.
arXiv Detail & Related papers (2023-05-22T17:58:04Z) - Bridging the Gap between Model Explanations in Partially Annotated
Multi-label Classification [85.76130799062379]
We study how false negative labels affect the model's explanation.
We propose to boost the attribution scores of the model trained with partial labels to make its explanation resemble that of the model trained with full labels.
arXiv Detail & Related papers (2023-04-04T14:00:59Z) - Label-enhanced Prototypical Network with Contrastive Learning for
Multi-label Few-shot Aspect Category Detection [17.228616743739412]
Multi-label aspect category detection allows a given review sentence to contain multiple aspect categories.
We propose a novel label-enhanced network (LPN) for multi-label few-shot aspect category detection.
arXiv Detail & Related papers (2022-06-14T02:37:44Z) - Resolving label uncertainty with implicit posterior models [71.62113762278963]
We propose a method for jointly inferring labels across a collection of data samples.
By implicitly assuming the existence of a generative model for which a differentiable predictor is the posterior, we derive a training objective that allows learning under weak beliefs.
arXiv Detail & Related papers (2022-02-28T18:09:44Z) - Exemplar Guided Active Learning [13.084183663366824]
We consider the problem of wisely using a limited budget to label a small subset of a large unlabeled dataset.
For any word, we have a set of candidate labels from a knowledge base, but the label set is not necessarily representative of what occurs in the data.
We describe an active learning approach that explicitly searches for rare classes by leveraging the contextual embedding spaces provided by modern language models.
arXiv Detail & Related papers (2020-11-02T20:01:39Z) - Exploiting Context for Robustness to Label Noise in Active Learning [47.341705184013804]
We address the problems of how a system can identify which of the queried labels are wrong and how a multi-class active learning system can be adapted to minimize the negative impact of label noise.
We construct a graphical representation of the unlabeled data to encode these relationships and obtain new beliefs on the graph when noisy labels are available.
This is demonstrated in three different applications: scene classification, activity classification, and document classification.
arXiv Detail & Related papers (2020-10-18T18:59:44Z) - Dynamic Semantic Matching and Aggregation Network for Few-shot Intent
Detection [69.2370349274216]
Few-shot Intent Detection is challenging due to the scarcity of available annotated utterances.
Semantic components are distilled from utterances via multi-head self-attention.
Our method provides a comprehensive matching measure to enhance representations of both labeled and unlabeled instances.
arXiv Detail & Related papers (2020-10-06T05:16:38Z) - Interaction Matching for Long-Tail Multi-Label Classification [57.262792333593644]
We present an elegant and effective approach for addressing limitations in existing multi-label classification models.
By performing soft n-gram interaction matching, we match labels with natural language descriptions.
arXiv Detail & Related papers (2020-05-18T15:27:55Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.