Augmenting NLP data to counter Annotation Artifacts for NLI Tasks

• URL: http://arxiv.org/abs/2302.04700v1
• Date: Thu, 9 Feb 2023 15:34:53 GMT
• Title: Augmenting NLP data to counter Annotation Artifacts for NLI Tasks
• Authors: Armaan Singh Bhullar
• Abstract summary: Large pre-trained NLP models achieve high performance on benchmark datasets but do not actually "solve" the underlying task. We explore this phenomenon by first using contrast and adversarial examples to understand limitations to the model's performance. We then propose a data augmentation technique to fix this bias and measure its effectiveness.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we explore Annotation Artifacts - the phenomena wherein large pre-trained NLP models achieve high performance on benchmark datasets but do not actually "solve" the underlying task and instead rely on some dataset artifacts (same across train, validation, and test sets) to figure out the right answer. We explore this phenomenon on the well-known Natural Language Inference task by first using contrast and adversarial examples to understand limitations to the model's performance and show one of the biases arising from annotation artifacts (the way training data was constructed by the annotators). We then propose a data augmentation technique to fix this bias and measure its effectiveness.

Related papers

• How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics [49.9329723199239]
  We propose a method for the automated creation of a challenging test set without relying on the manual construction of artificial and unrealistic examples. We categorize the test set of popular NLI datasets into three difficulty levels by leveraging methods that exploit training dynamics. When our characterization method is applied to the training set, models trained with only a fraction of the data achieve comparable performance to those trained on the full dataset.
  arXiv  Detail & Related papers  (2024-10-04T13:39:21Z)
• SINDER: Repairing the Singular Defects of DINOv2 [61.98878352956125]
  Vision Transformer models trained on large-scale datasets often exhibit artifacts in the patch token they extract. We propose a novel fine-tuning smooth regularization that rectifies structural deficiencies using only a small dataset.
  arXiv  Detail & Related papers  (2024-07-23T20:34:23Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• Improving the Adversarial Robustness of NLP Models by Information Bottleneck [112.44039792098579]
  Non-robust features can be easily manipulated by adversaries to fool NLP models. In this study, we explore the feasibility of capturing task-specific robust features, while eliminating the non-robust ones by using the information bottleneck theory. We show that the models trained with our information bottleneck-based method are able to achieve a significant improvement in robust accuracy.
  arXiv  Detail & Related papers  (2022-06-11T12:12:20Z)
• Robust Task-Oriented Dialogue Generation with Contrastive Pre-training and Adversarial Filtering [17.7709632238066]
  Data artifacts incentivize machine learning models to learn non-transferable generalizations. We investigate whether popular datasets such as MultiWOZ contain such data artifacts. We propose a contrastive learning based framework to encourage the model to ignore these cues and focus on learning generalisable patterns.
  arXiv  Detail & Related papers  (2022-05-20T03:13:02Z)
• Task-guided Disentangled Tuning for Pretrained Language Models [16.429787408467703]
  We propose Task-guided Disentangled Tuning (TDT) for pretrained language models (PLMs) TDT enhances the generalization of representations by disentangling task-relevant signals from entangled representations. Experimental results on GLUE and CLUE benchmarks show that TDT gives consistently better results than fine-tuning with different PLMs.
  arXiv  Detail & Related papers  (2022-03-22T03:11:39Z)
• Combining Feature and Instance Attribution to Detect Artifacts [62.63504976810927]
  We propose methods to facilitate identification of training data artifacts. We show that this proposed training-feature attribution approach can be used to uncover artifacts in training data. We execute a small user study to evaluate whether these methods are useful to NLP researchers in practice.
  arXiv  Detail & Related papers  (2021-07-01T09:26:13Z)
• DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks [88.62288327934499]
  We propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-11-03T07:49:15Z)
• Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data [27.738670027154555]
  Counterfactual augmentation of natural language understanding data does not appear to be an effective way of collecting training data. We build upon this work by using English natural language inference data to test model generalization and robustness.
  arXiv  Detail & Related papers  (2020-10-09T18:44:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.