To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering

• URL: http://arxiv.org/abs/2212.10381v1
• Date: Tue, 20 Dec 2022 16:06:09 GMT
• Title: To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering
• Authors: Dheeru Dua, Emma Strubell, Sameer Singh, Pat Verga
• Abstract summary: We study end-to-end model performance of open-domain question answering (ODQA) We find that not only do models fail to generalize, but high retrieval scores often still yield poor answer prediction accuracy. We propose and evaluate several intervention methods which improve end-to-end answer F1 score by up to 24 points.
• Score: 46.403929561360485
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in open-domain question answering (ODQA) have demonstrated impressive accuracy on standard Wikipedia style benchmarks. However, it is less clear how robust these models are and how well they perform when applied to real-world applications in drastically different domains. While there has been some work investigating how well ODQA models perform when tested for out-of-domain (OOD) generalization, these studies have been conducted only under conservative shifts in data distribution and typically focus on a single component (ie. retrieval) rather than an end-to-end system. In response, we propose a more realistic and challenging domain shift evaluation setting and, through extensive experiments, study end-to-end model performance. We find that not only do models fail to generalize, but high retrieval scores often still yield poor answer prediction accuracy. We then categorize different types of shifts and propose techniques that, when presented with a new dataset, predict if intervention methods are likely to be successful. Finally, using insights from this analysis, we propose and evaluate several intervention methods which improve end-to-end answer F1 score by up to 24 points.

Related papers

• Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization [64.62570402941387]
  We use a single test sample to adapt multi-modal prompts at test time by minimizing the feature distribution shift to bridge the gap in the test domain. Our method improves zero-shot top- 1 accuracy beyond existing prompt-learning techniques, with a 3.08% improvement over the baseline MaPLe.
  arXiv  Detail & Related papers  (2023-11-02T17:59:32Z)
• Activate and Reject: Towards Safe Domain Generalization under Category Shift [71.95548187205736]
  We study a practical problem of Domain Generalization under Category Shift (DGCS) It aims to simultaneously detect unknown-class samples and classify known-class samples in the target domains. Compared to prior DG works, we face two new challenges: 1) how to learn the concept of unknown'' during training with only source known-class samples, and 2) how to adapt the source-trained model to unseen environments.
  arXiv  Detail & Related papers  (2023-10-07T07:53:12Z)
• Better Practices for Domain Adaptation [62.70267990659201]
  Domain adaptation (DA) aims to provide frameworks for adapting models to deployment data without using labels. Unclear validation protocol for DA has led to bad practices in the literature. We show challenges across all three branches of domain adaptation methodology.
  arXiv  Detail & Related papers  (2023-09-07T17:44:18Z)
• Test-time Training for Data-efficient UCDR [22.400837122986175]
  Universal Cross-domain Retrieval protocol is a pioneer in this field. In this work, we explore the generalized retrieval problem in a data-efficient manner.
  arXiv  Detail & Related papers  (2022-08-19T07:50:04Z)
• IDANI: Inference-time Domain Adaptation via Neuron-level Interventions [24.60778570114818]
  We propose a new approach for domain adaptation (DA), using neuron-level interventions. We modify the representation of each test example in specific neurons, resulting in a counterfactual example from the source domain. Our experiments show that our method improves performance on unseen domains.
  arXiv  Detail & Related papers  (2022-06-01T06:39:28Z)
• ADATIME: A Benchmarking Suite for Domain Adaptation on Time Series Data [20.34427953468868]
  Unsupervised domain adaptation methods aim to generalize well on unlabeled test data that may have a different distribution from the training data. Existing works on time series domain adaptation suffer from inconsistencies in evaluation schemes, datasets, and backbone neural network architectures. We develop a benchmarking evaluation suite (AdaTime) to systematically and fairly evaluate different domain adaptation methods on time series data.
  arXiv  Detail & Related papers  (2022-03-15T23:55:05Z)
• Adaptive Risk Minimization: Learning to Adapt to Domain Shift [109.87561509436016]
  A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution. In this work, we consider the problem setting of domain generalization, where the training data are structured into domains and there may be multiple test time shifts. We introduce the framework of adaptive risk minimization (ARM), in which models are directly optimized for effective adaptation to shift by learning to adapt on the training domains.
  arXiv  Detail & Related papers  (2020-07-06T17:59:30Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)
• A simple baseline for domain adaptation using rotation prediction [17.539027866457673]
  The goal is to adapt a model trained in one domain to another domain with scarce annotated data. We propose a simple yet effective method based on self-supervised learning. Our simple method achieves state-of-the-art results on semi-supervised domain adaptation on DomainNet dataset.
  arXiv  Detail & Related papers  (2019-12-26T17:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.