Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering

• URL: http://arxiv.org/abs/2310.17490v3
• Date: Tue, 14 Nov 2023 06:49:33 GMT
• Title: Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
• Authors: Sukmin Cho, Jeongyeon Seo, Soyeong Jeong, Jong C. Park
• Abstract summary: Large language models (LLMs) enable zero-shot approaches in open-domain question answering (ODQA) This study aims at the feasibility of a zero-shot reader that addresses the challenges of computational cost and the need for labeled data.
• Score: 10.794156033638984
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) enable zero-shot approaches in open-domain question answering (ODQA), yet with limited advancements as the reader is compared to the retriever. This study aims at the feasibility of a zero-shot reader that addresses the challenges of computational cost and the need for labeled data. We find that LLMs are distracted due to irrelevant documents in the retrieved set and the overconfidence of the generated answers when they are exploited as zero-shot readers. To tackle these problems, we mitigate the impact of such documents via Distraction-aware Answer Selection (DAS) with a negation-based instruction and score adjustment for proper answer selection. Experimental results show that our approach successfully handles distraction across diverse scenarios, enhancing the performance of zero-shot readers. Furthermore, unlike supervised readers struggling with unseen data, zero-shot readers demonstrate outstanding transferability without any training.

Related papers

• Calibrated Decision-Making through LLM-Assisted Retrieval [21.14260990790119]
  When large language models confidently provide incorrect information, it can lead humans to make suboptimal decisions. We propose Calibrated Retrieval-Augmented Generation (CalibRAG) to ensure that decisions informed by retrieved documents are well-calibrated.
  arXiv  Detail & Related papers  (2024-10-28T06:41:05Z)
• DORY: Deliberative Prompt Recovery for LLM [11.988508965818767]
  Deliberative PrOmpt RecoverY (DORY) is a novel approach that leverages uncertainty to recover prompts accurately. DORY involves reconstructing drafts from outputs, refining these with hints, and filtering out noise based on uncertainty. Our evaluation shows that DORY outperforms existing baselines, improving performance by approximately 10.82%.
  arXiv  Detail & Related papers  (2024-05-31T07:51:16Z)
• Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore [51.65730053591696]
  We propose a simple but effective black-box zero-shot detection approach. It is predicated on the observation that human-written texts typically contain more grammatical errors than LLM-generated texts. Our method achieves an average AUROC of 98.7% and shows strong robustness against paraphrase and adversarial perturbation attacks.
  arXiv  Detail & Related papers  (2024-05-07T12:57:01Z)
• Answerability in Retrieval-Augmented Open-Domain Question Answering [17.177439885871788]
  Open-Domain Question Answering (ODQA) retrieval systems can exhibit sub-optimal behavior, providing text excerpts with varying degrees of irrelevance. Previous attempts to address this gap have relied on a simplistic approach of pairing questions with random text excerpts.
  arXiv  Detail & Related papers  (2024-03-03T09:55:35Z)
• Revisiting Large Language Models as Zero-shot Relation Extractors [8.953462875381888]
  Relation extraction (RE) consistently involves a certain degree of labeled or unlabeled data even if under zero-shot setting. Recent studies have shown that large language models (LLMs) transfer well to new tasks out-of-the-box simply given a natural language prompt. This work focuses on the study of exploring LLMs as zero-shot relation extractors.
  arXiv  Detail & Related papers  (2023-10-08T06:17:39Z)
• SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning [55.76083560152823]
  SelfCheck is a general-purpose zero-shot verification schema for recognizing errors in step-by-step reasoning. We test SelfCheck on three datasets (GSM8K, MathQA, and MATH) and find that it successfully recognizes errors and, in turn, increases final answer accuracies.
  arXiv  Detail & Related papers  (2023-08-01T10:31:36Z)
• Evidentiality-aware Retrieval for Overcoming Abstractiveness in Open-Domain Question Answering [29.00167886463793]
  We propose Evidentiality-Aware Passage Retrieval (EADPR) to learn to discriminate evidence passages from distractors. We conduct extensive experiments to validate the effectiveness of our proposed method on multiple abstractive ODQA tasks.
  arXiv  Detail & Related papers  (2023-04-06T12:42:37Z)
• Self-Prompting Large Language Models for Zero-Shot Open-Domain QA [67.08732962244301]
  Open-Domain Question Answering (ODQA) aims to answer questions without explicitly providing background documents. This task becomes notably challenging in a zero-shot setting where no data is available to train tailored retrieval-reader models. We propose a Self-Prompting framework to explicitly utilize the massive knowledge encoded in the parameters of Large Language Models.
  arXiv  Detail & Related papers  (2022-12-16T18:23:43Z)
• ALLSH: Active Learning Guided by Local Sensitivity and Hardness [98.61023158378407]
  We propose to retrieve unlabeled samples with a local sensitivity and hardness-aware acquisition function. Our method achieves consistent gains over the commonly used active learning strategies in various classification tasks.
  arXiv  Detail & Related papers  (2022-05-10T15:39:11Z)
• LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval [55.097573036580066]
  Experimental results show that LaPraDoR achieves state-of-the-art performance compared with supervised dense retrieval models. Compared to re-ranking, our lexicon-enhanced approach can be run in milliseconds (22.5x faster) while achieving superior performance.
  arXiv  Detail & Related papers  (2022-03-11T18:53:12Z)
• Retrospective Reader for Machine Reading Comprehension [90.6069071495214]
  Machine reading comprehension (MRC) is an AI challenge that requires machine to determine the correct answers to questions based on a given passage. When unanswerable questions are involved in the MRC task, an essential verification module called verifier is especially required in addition to the encoder. This paper devotes itself to exploring better verifier design for the MRC task with unanswerable questions.
  arXiv  Detail & Related papers  (2020-01-27T11:14:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.