Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions

• URL: http://arxiv.org/abs/2205.00415v3
• Date: Wed, 20 Mar 2024 03:23:11 GMT
• Title: Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions
• Authors: Mihir Parmar, Swaroop Mishra, Mor Geva, Chitta Baral,
• Abstract summary: We study a form of bias, termed instruction bias, in 14 recent NLU benchmarks. We show that instruction examples often exhibit concrete patterns, which are propagated by crowdworkers to the collected data.
• Score: 71.5668415104079
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, progress in NLU has been driven by benchmarks. These benchmarks are typically collected by crowdsourcing, where annotators write examples based on annotation instructions crafted by dataset creators. In this work, we hypothesize that annotators pick up on patterns in the crowdsourcing instructions, which bias them to write many similar examples that are then over-represented in the collected data. We study this form of bias, termed instruction bias, in 14 recent NLU benchmarks, showing that instruction examples often exhibit concrete patterns, which are propagated by crowdworkers to the collected data. This extends previous work (Geva et al., 2019) and raises a new concern of whether we are modeling the dataset creator's instructions, rather than the task. Through a series of experiments, we show that, indeed, instruction bias can lead to overestimation of model performance, and that models struggle to generalize beyond biases originating in the crowdsourcing instructions. We further analyze the influence of instruction bias in terms of pattern frequency and model size, and derive concrete recommendations for creating future NLU benchmarks.

Related papers

• Context-Parametric Inversion: Why Instruction Finetuning May Not Actually Improve Context Reliance [68.56701216210617]
  In-principle, one would expect models to adapt to the user context better after instruction finetuning. We observe a surprising failure mode: during instruction tuning, the context reliance under knowledge conflicts initially increases as expected, but then gradually decreases.
  arXiv  Detail & Related papers  (2024-10-14T17:57:09Z)
• Benchmarking Benchmark Leakage in Large Language Models [24.015208839742343]
  We introduce a detection pipeline utilizing Perplexity and N-gram accuracy, two simple and scalable metrics that gauge a model's prediction precision on benchmark. We reveal substantial instances of training even test set misuse, resulting in potentially unfair comparisons. We propose the "Benchmark Transparency Card" to encourage clear documentation of benchmark utilization.
  arXiv  Detail & Related papers  (2024-04-29T16:05:36Z)
• IBADR: an Iterative Bias-Aware Dataset Refinement Framework for Debiasing NLU models [52.03761198830643]
  We propose IBADR, an Iterative Bias-Aware dataset Refinement framework. We first train a shallow model to quantify the bias degree of samples in the pool. Then, we pair each sample with a bias indicator representing its bias degree, and use these extended samples to train a sample generator. In this way, this generator can effectively learn the correspondence relationship between bias indicators and samples.
  arXiv  Detail & Related papers  (2023-11-01T04:50:38Z)
• Fairness-guided Few-shot Prompting for Large Language Models [93.05624064699965]
  In-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats. We introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes. We propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning.
  arXiv  Detail & Related papers  (2023-03-23T12:28:25Z)
• Augmenting NLP data to counter Annotation Artifacts for NLI Tasks [0.0]
  Large pre-trained NLP models achieve high performance on benchmark datasets but do not actually "solve" the underlying task. We explore this phenomenon by first using contrast and adversarial examples to understand limitations to the model's performance. We then propose a data augmentation technique to fix this bias and measure its effectiveness.
  arXiv  Detail & Related papers  (2023-02-09T15:34:53Z)
• Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data [27.738670027154555]
  Counterfactual augmentation of natural language understanding data does not appear to be an effective way of collecting training data. We build upon this work by using English natural language inference data to test model generalization and robustness.
  arXiv  Detail & Related papers  (2020-10-09T18:44:02Z)
• Towards Debiasing NLU Models from Unknown Biases [70.31427277842239]
  NLU models often exploit biases to achieve high dataset-specific performance without properly learning the intended task. We present a self-debiasing framework that prevents models from mainly utilizing biases without knowing them in advance.
  arXiv  Detail & Related papers  (2020-09-25T15:49:39Z)
• Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance [70.31427277842239]
  We introduce a novel debiasing method called confidence regularization. It discourages models from exploiting biases while enabling them to receive enough incentive to learn from all the training examples. We evaluate our method on three NLU tasks and show that, in contrast to its predecessors, it improves the performance on out-of-distribution datasets.
  arXiv  Detail & Related papers  (2020-05-01T11:22:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.