Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation

• URL: http://arxiv.org/abs/2304.12443v1
• Date: Mon, 24 Apr 2023 20:45:09 GMT
• Title: Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation
• Authors: Nan-Jiang Jiang, Chenhao Tan, Marie-Catherine de Marneffe
• Abstract summary: We create the first ecologically valid explanation dataset with diverse reasoning, LiveNLI. LiveNLI contains annotators' highlights and free-text explanations for the label(s) of their choice for 122 English Natural Language Inference items. We used its explanations for chain-of-thought prompting, and found there is still room for improvement in GPT-3's ability to predict label distribution with in-context learning.
• Score: 18.161206115232066
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Human label variation (Plank 2022), or annotation disagreement, exists in many natural language processing (NLP) tasks. To be robust and trusted, NLP models need to identify such variation and be able to explain it. To this end, we created the first ecologically valid explanation dataset with diverse reasoning, LiveNLI. LiveNLI contains annotators' highlights and free-text explanations for the label(s) of their choice for 122 English Natural Language Inference items, each with at least 10 annotations. We used its explanations for chain-of-thought prompting, and found there is still room for improvement in GPT-3's ability to predict label distribution with in-context learning.

Related papers

• TAGExplainer: Narrating Graph Explanations for Text-Attributed Graph Learning Models [14.367754016281934]
  This paper presents TAGExplainer, the first method designed to generate natural language explanations for TAG learning. To address the lack of annotated ground truth explanations in real-world scenarios, we propose first generating pseudo-labels that capture the model's decisions from saliency-based explanations. The high-quality pseudo-labels are finally utilized to train an end-to-end explanation generator model.
  arXiv  Detail & Related papers  (2024-10-20T03:55:46Z)
• A Taxonomy of Ambiguity Types for NLP [53.10379645698917]
  We propose a taxonomy of ambiguity types as seen in English to facilitate NLP analysis. Our taxonomy can help make meaningful splits in language ambiguity data, allowing for more fine-grained assessments of both datasets and model performance.
  arXiv  Detail & Related papers  (2024-03-21T01:47:22Z)
• Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions [50.92702206798324]
  We propose a generative prompting framework for zero-shot text classification. GEN-Z measures the LM likelihood of input text conditioned on natural language descriptions of labels. We show that zero-shot classification with simple contextualization of the data source consistently outperforms both zero-shot and few-shot baselines.
  arXiv  Detail & Related papers  (2023-11-13T07:12:57Z)
• Ecologically Valid Explanations for Label Variation in NLI [27.324994764803808]
  We build LiveNLI, an English dataset of 1,415 ecologically valid explanations (annotators explain the NLI labels they chose) for 122 MNLI items. LiveNLI explanations confirm that people can systematically vary on their interpretation and highlight within-label variation. This suggests that explanations are crucial for navigating label interpretations in general.
  arXiv  Detail & Related papers  (2023-10-20T22:52:19Z)
• FLamE: Few-shot Learning from Natural Language Explanations [12.496665033682202]
  We present FLamE, a framework for learning from natural language explanations. Experiments on natural language inference demonstrate effectiveness over strong baselines. Human evaluation surprisingly reveals that the majority of generated explanations does not adequately justify classification decisions.
  arXiv  Detail & Related papers  (2023-06-13T18:01:46Z)
• Symbol tuning improves in-context learning in language models [144.58397538701803]
  We present symbol tuning - finetuning language models on in-context input-label pairs. Symbol tuning leverages the intuition that when a model cannot use instructions or natural language labels to figure out a task, it must instead do so by learning the input-label mappings. We show that symbol tuning boosts performance on unseen in-context learning tasks and is much more robust to underspecified prompts.
  arXiv  Detail & Related papers  (2023-05-15T01:59:58Z)
• Investigating Reasons for Disagreement in Natural Language Inference [9.002603216969154]
  We develop a taxonomy of disagreement sources spanning 3 high-level classes. Some disagreements are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts. We explore two modeling approaches for detecting items with potential disagreement.
  arXiv  Detail & Related papers  (2022-09-07T18:01:39Z)
• VALUE: Understanding Dialect Disparity in NLU [50.35526025326337]
  We construct rules for 11 features of African American Vernacular English (AAVE) We recruit fluent AAVE speakers to validate each feature transformation via linguistic acceptability judgments. Experiments show that these new dialectal features can lead to a drop in model performance.
  arXiv  Detail & Related papers  (2022-04-06T18:30:56Z)
• Interpreting Language Models with Contrastive Explanations [99.7035899290924]
  Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics. Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding. We show that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena.
  arXiv  Detail & Related papers  (2022-02-21T18:32:24Z)
• ALICE: Active Learning with Contrastive Natural Language Explanations [69.03658685761538]
  We propose Active Learning with Contrastive Explanations (ALICE) to improve data efficiency in learning. ALICE learns to first use active learning to select the most informative pairs of label classes to elicit contrastive natural language explanations. It extracts knowledge from these explanations using a semantically extracted knowledge.
  arXiv  Detail & Related papers  (2020-09-22T01:02:07Z)
• NILE : Natural Language Inference with Faithful Natural Language Explanations [10.074153632701952]
  We propose Natural-language Inference over Label-specific Explanations (NILE) NILE is a novel NLI method which utilizes auto-generated label-specific explanations to produce labels along with its faithful explanation. We discuss the faithfulness of NILE's explanations in terms of sensitivity of the decisions to the corresponding explanations.
  arXiv  Detail & Related papers  (2020-05-25T13:56:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.