Towards Better Generalization in Open-Domain Question Answering by Mitigating Context Memorization

• URL: http://arxiv.org/abs/2404.01652v1
• Date: Tue, 2 Apr 2024 05:44:50 GMT
• Title: Towards Better Generalization in Open-Domain Question Answering by Mitigating Context Memorization
• Authors: Zixuan Zhang, Revanth Gangi Reddy, Kevin Small, Tong Zhang, Heng Ji,
• Abstract summary: Open-domain Question Answering (OpenQA) aims at answering factual questions with an external large-scale knowledge corpus. It is still unclear how well an OpenQA model can transfer to completely new knowledge domains. We introduce Corpus-Invariant Tuning (CIT), a simple but effective training strategy, to mitigate the knowledge over-memorization.
• Score: 67.92796510359595
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-domain Question Answering (OpenQA) aims at answering factual questions with an external large-scale knowledge corpus. However, real-world knowledge is not static; it updates and evolves continually. Such a dynamic characteristic of knowledge poses a vital challenge for these models, as the trained models need to constantly adapt to the latest information to make sure that the answers remain accurate. In addition, it is still unclear how well an OpenQA model can transfer to completely new knowledge domains. In this paper, we investigate the generalization performance of a retrieval-augmented QA model in two specific scenarios: 1) adapting to updated versions of the same knowledge corpus; 2) switching to completely different knowledge domains. We observe that the generalization challenges of OpenQA models stem from the reader's over-reliance on memorizing the knowledge from the external corpus, which hinders the model from generalizing to a new knowledge corpus. We introduce Corpus-Invariant Tuning (CIT), a simple but effective training strategy, to mitigate the knowledge over-memorization by controlling the likelihood of retrieved contexts during training. Extensive experimental results on multiple OpenQA benchmarks show that CIT achieves significantly better generalizability without compromising the model's performance in its original corpus and domain.

Related papers

• Retriever-and-Memory: Towards Adaptive Note-Enhanced Retrieval-Augmented Generation [72.70046559930555]
  We propose a generic RAG approach called Adaptive Note-Enhanced RAG (Adaptive-Note) for complex QA tasks. Specifically, Adaptive-Note introduces an overarching view of knowledge growth, iteratively gathering new information in the form of notes. In addition, we employ an adaptive, note-based stop-exploration strategy to decide "what to retrieve and when to stop" to encourage sufficient knowledge exploration.
  arXiv  Detail & Related papers  (2024-10-11T14:03:29Z)
• Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answering [11.183845003492964]
  We use Passage Retrieval (DPR) to retrieve related knowledge to help the model answer questions. DPR conduct retrieving in natural language space, which may not ensure comprehensive acquisition of image information. We propose a novel framework that leverages the visual-language model to select the key knowledge retrieved by DPR and answer questions.
  arXiv  Detail & Related papers  (2024-04-22T07:44:20Z)
• DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text [73.68051228972024]
  Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when relying on their internal knowledge. Retrieval-augmented LLMs have emerged as a potential solution to ground LLMs in external knowledge.
  arXiv  Detail & Related papers  (2023-10-31T04:37:57Z)
• Self-Knowledge Guided Retrieval Augmentation for Large Language Models [59.771098292611846]
  Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. Self-Knowledge guided Retrieval augmentation (SKR) is a simple yet effective method which can let LLMs refer to the questions they have previously encountered.
  arXiv  Detail & Related papers  (2023-10-08T04:22:33Z)
• KAT: A Knowledge Augmented Transformer for Vision-and-Language [56.716531169609915]
  We propose a novel model - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an end to end encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. An additional benefit of explicit knowledge integration is seen in improved interpretability of model predictions in our analysis.
  arXiv  Detail & Related papers  (2021-12-16T04:37:10Z)
• KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA [107.7091094498848]
  One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image. In this work we study open-domain knowledge, the setting when the knowledge required to answer a question is not given/annotated, neither at training nor test time. We tap into two types of knowledge representations and reasoning. First, implicit knowledge which can be learned effectively from unsupervised language pre-training and supervised training data with transformer-based models.
  arXiv  Detail & Related papers  (2020-12-20T20:13:02Z)
• REALM: Retrieval-Augmented Language Model Pre-Training [37.3178586179607]
  We augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA)
  arXiv  Detail & Related papers  (2020-02-10T18:40:59Z)
• What Does My QA Model Know? Devising Controlled Probes using Expert Knowledge [36.13528043657398]
  We investigate whether state-of-the-art QA models have general knowledge about word definitions and general taxonomic reasoning. We use a methodology for automatically building datasets from various types of expert knowledge. Our evaluation confirms that transformer-based QA models are already predisposed to recognize certain types of structural lexical knowledge.
  arXiv  Detail & Related papers  (2019-12-31T15:05:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.