Which Shortcut Solution Do Question Answering Models Prefer to Learn?

• URL: http://arxiv.org/abs/2211.16220v1
• Date: Tue, 29 Nov 2022 13:57:59 GMT
• Title: Which Shortcut Solution Do Question Answering Models Prefer to Learn?
• Authors: Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
• Abstract summary: Question answering (QA) models for reading comprehension tend to learn shortcut solutions rather than the solutions intended by QA datasets. We show that shortcuts that exploit answer positions and word-label correlations are preferentially learned for extractive and multiple-choice QA. We experimentally show that the learnability of shortcuts can be utilized to construct an effective QA training set.
• Score: 38.36299280464046
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Question answering (QA) models for reading comprehension tend to learn shortcut solutions rather than the solutions intended by QA datasets. QA models that have learned shortcut solutions can achieve human-level performance in shortcut examples where shortcuts are valid, but these same behaviors degrade generalization potential on anti-shortcut examples where shortcuts are invalid. Various methods have been proposed to mitigate this problem, but they do not fully take the characteristics of shortcuts themselves into account. We assume that the learnability of shortcuts, i.e., how easy it is to learn a shortcut, is useful to mitigate the problem. Thus, we first examine the learnability of the representative shortcuts on extractive and multiple-choice QA datasets. Behavioral tests using biased training sets reveal that shortcuts that exploit answer positions and word-label correlations are preferentially learned for extractive and multiple-choice QA, respectively. We find that the more learnable a shortcut is, the flatter and deeper the loss landscape is around the shortcut solution in the parameter space. We also find that the availability of the preferred shortcuts tends to make the task easier to perform from an information-theoretic viewpoint. Lastly, we experimentally show that the learnability of shortcuts can be utilized to construct an effective QA training set; the more learnable a shortcut is, the smaller the proportion of anti-shortcut examples required to achieve comparable performance on shortcut and anti-shortcut examples. We claim that the learnability of shortcuts should be considered when designing mitigation methods.

Related papers

• Exposing Image Classifier Shortcuts with Counterfactual Frequency (CoF) Tables [0.0]
  'Shortcuts' are easy-to-learn patterns from the training data that fail to generalise to new data. Examples include the use of a copyright watermark to recognise horses, snowy background to recognise huskies, or ink markings to detect malignant skin lesions. We introduce Counterfactual Frequency tables, a novel approach that aggregates instance-based explanations into global insights.
  arXiv  Detail & Related papers  (2024-05-24T15:58:02Z)
• Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model [86.9619638550683]
  Vision-language foundation models have exhibited remarkable success across a multitude of downstream tasks due to their scalability on extensive image-text paired data. However, these models display significant limitations when applied to downstream tasks, such as fine-grained image classification, as a result of decision shortcuts''
  arXiv  Detail & Related papers  (2024-03-01T09:01:53Z)
• A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others [48.11387483887109]
  Key to advancing reliability of vision systems is understanding whether existing methods can overcome multiple shortcuts or struggle in a Whac-A-Mole game. We find computer vision models, including large foundation models, struggle when multiple shortcuts are present. We propose Last Layer Ensemble, a simple-yet-effective method to mitigate multiple shortcuts without Whac-A-Mole behavior.
  arXiv  Detail & Related papers  (2022-12-09T18:59:57Z)
• Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA [53.45074798673808]
  VQA models are prone to learn the shortcut solution formed by dataset biases rather than the intended solution. We propose a new dataset that considers varying types of shortcuts by constructing different distribution shifts in multiple OOD test sets. Our benchmark provides a more rigorous and comprehensive testbed for shortcut learning in VQA.
  arXiv  Detail & Related papers  (2022-10-10T13:39:08Z)
• Why Machine Reading Comprehension Models Learn Shortcuts? [56.629192589376046]
  We argue that larger proportion of shortcut questions in training data make models rely on shortcut tricks excessively. A thorough empirical analysis shows that MRC models tend to learn shortcut questions earlier than challenging questions.
  arXiv  Detail & Related papers  (2021-06-02T08:43:12Z)
• Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering [42.120558318437475]
  Shortcut learning happens when a model exploits spurious statistical regularities to produce correct answers but does not deploy the desired behavior. We introduce an evaluation methodology for visual question answering (VQA) to better diagnose cases of shortcut learning.
  arXiv  Detail & Related papers  (2021-04-07T14:28:22Z)
• Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU models [53.36605766266518]
  We show that trained NLU models have strong preference for features located at the head of the long-tailed distribution. We propose a shortcut mitigation framework, to suppress the model from making overconfident predictions for samples with large shortcut degree.
  arXiv  Detail & Related papers  (2021-03-11T19:39:56Z)
• Information-Preserving Contrastive Learning for Self-Supervised Representations [42.74927142208983]
  Contrastive learning is very effective at learning useful representations without supervision. It may learn a shortcut that is irrelevant to the downstream task, and discard relevant information. This paper presents information-preserving contrastive learning (IPCL), a new framework for unsupervised representation learning.
  arXiv  Detail & Related papers  (2020-12-17T22:46:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.