Ecologically Valid Explanations for Label Variation in NLI

• URL: http://arxiv.org/abs/2310.13850v1
• Date: Fri, 20 Oct 2023 22:52:19 GMT
• Title: Ecologically Valid Explanations for Label Variation in NLI
• Authors: Nan-Jiang Jiang, Chenhao Tan, Marie-Catherine de Marneffe
• Abstract summary: We build LiveNLI, an English dataset of 1,415 ecologically valid explanations (annotators explain the NLI labels they chose) for 122 MNLI items. LiveNLI explanations confirm that people can systematically vary on their interpretation and highlight within-label variation. This suggests that explanations are crucial for navigating label interpretations in general.
• Score: 27.324994764803808
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Human label variation, or annotation disagreement, exists in many natural language processing (NLP) tasks, including natural language inference (NLI). To gain direct evidence of how NLI label variation arises, we build LiveNLI, an English dataset of 1,415 ecologically valid explanations (annotators explain the NLI labels they chose) for 122 MNLI items (at least 10 explanations per item). The LiveNLI explanations confirm that people can systematically vary on their interpretation and highlight within-label variation: annotators sometimes choose the same label for different reasons. This suggests that explanations are crucial for navigating label interpretations in general. We few-shot prompt large language models to generate explanations but the results are inconsistent: they sometimes produces valid and informative explanations, but it also generates implausible ones that do not support the label, highlighting directions for improvement.

Related papers

• VariErr NLI: Separating Annotation Error from Human Label Variation [23.392480595432676]
  We introduce a systematic methodology and a new dataset, VariErr (variation versus error) VariErr contains 7,732 validity judgments on 1,933 explanations for 500 re-annotated MNLI items. We find that state-of-the-art AED methods significantly underperform GPTs and humans.
  arXiv  Detail & Related papers  (2024-03-04T10:57:14Z)
• Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions [50.92702206798324]
  We propose a generative prompting framework for zero-shot text classification. GEN-Z measures the LM likelihood of input text conditioned on natural language descriptions of labels. We show that zero-shot classification with simple contextualization of the data source consistently outperforms both zero-shot and few-shot baselines.
  arXiv  Detail & Related papers  (2023-11-13T07:12:57Z)
• Symbol tuning improves in-context learning in language models [144.58397538701803]
  We present symbol tuning - finetuning language models on in-context input-label pairs. Symbol tuning leverages the intuition that when a model cannot use instructions or natural language labels to figure out a task, it must instead do so by learning the input-label mappings. We show that symbol tuning boosts performance on unseen in-context learning tasks and is much more robust to underspecified prompts.
  arXiv  Detail & Related papers  (2023-05-15T01:59:58Z)
• We're Afraid Language Models Aren't Modeling Ambiguity [136.8068419824318]
  Managing ambiguity is a key part of human language understanding. We characterize ambiguity in a sentence by its effect on entailment relations with another sentence. We show that a multilabel NLI model can flag political claims in the wild that are misleading due to ambiguity.
  arXiv  Detail & Related papers  (2023-04-27T17:57:58Z)
• Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation [18.161206115232066]
  We create the first ecologically valid explanation dataset with diverse reasoning, LiveNLI. LiveNLI contains annotators' highlights and free-text explanations for the label(s) of their choice for 122 English Natural Language Inference items. We used its explanations for chain-of-thought prompting, and found there is still room for improvement in GPT-3's ability to predict label distribution with in-context learning.
  arXiv  Detail & Related papers  (2023-04-24T20:45:09Z)
• Investigating Reasons for Disagreement in Natural Language Inference [9.002603216969154]
  We develop a taxonomy of disagreement sources spanning 3 high-level classes. Some disagreements are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts. We explore two modeling approaches for detecting items with potential disagreement.
  arXiv  Detail & Related papers  (2022-09-07T18:01:39Z)
• VALUE: Understanding Dialect Disparity in NLU [50.35526025326337]
  We construct rules for 11 features of African American Vernacular English (AAVE) We recruit fluent AAVE speakers to validate each feature transformation via linguistic acceptability judgments. Experiments show that these new dialectal features can lead to a drop in model performance.
  arXiv  Detail & Related papers  (2022-04-06T18:30:56Z)
• Interpreting Language Models with Contrastive Explanations [99.7035899290924]
  Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics. Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding. We show that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena.
  arXiv  Detail & Related papers  (2022-02-21T18:32:24Z)
• Contrastive Explanations for Model Interpretability [77.92370750072831]
  We propose a methodology to produce contrastive explanations for classification models. Our method is based on projecting model representation to a latent space. Our findings shed light on the ability of label-contrastive explanations to provide a more accurate and finer-grained interpretability of a model's decision.
  arXiv  Detail & Related papers  (2021-03-02T00:36:45Z)
• LIREx: Augmenting Language Inference with Relevant Explanation [1.4780878458667916]
  Natural language explanations (NLEs) are a form of data annotation in which annotators identify rationales when assigning labels to data instances. NLEs have been shown to capture human reasoning better, but not as beneficial for natural language inference. We propose a novel framework, LIREx, that incorporates both a rationale-enabled explanation generator and an instance selector to select only relevant NLEs.
  arXiv  Detail & Related papers  (2020-12-16T18:49:29Z)
• NILE : Natural Language Inference with Faithful Natural Language Explanations [10.074153632701952]
  We propose Natural-language Inference over Label-specific Explanations (NILE) NILE is a novel NLI method which utilizes auto-generated label-specific explanations to produce labels along with its faithful explanation. We discuss the faithfulness of NILE's explanations in terms of sensitivity of the decisions to the corresponding explanations.
  arXiv  Detail & Related papers  (2020-05-25T13:56:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.