Corpus-informed Retrieval Augmented Generation of Clarifying Questions

• URL: http://arxiv.org/abs/2409.18575v1
• Date: Fri, 27 Sep 2024 09:20:42 GMT
• Title: Corpus-informed Retrieval Augmented Generation of Clarifying Questions
• Authors: Antonios Minas Krasakis, Andrew Yates, Evangelos Kanoulas,
• Abstract summary: This study aims to develop models that generate corpus informed clarifying questions for web search. In current datasets search intents are largely unsupported by the corpus, which is problematic both for training and evaluation. We propose dataset augmentation methods that align the ground truth clarifications with the retrieval corpus.
• Score: 23.123116796159717
• License:
• Abstract: This study aims to develop models that generate corpus informed clarifying questions for web search, in a way that ensures the questions align with the available information in the retrieval corpus. We demonstrate the effectiveness of Retrieval Augmented Language Models (RAG) in this process, emphasising their ability to (i) jointly model the user query and retrieval corpus to pinpoint the uncertainty and ask for clarifications end-to-end and (ii) model more evidence documents, which can be used towards increasing the breadth of the questions asked. However, we observe that in current datasets search intents are largely unsupported by the corpus, which is problematic both for training and evaluation. This causes question generation models to ``hallucinate'', ie. suggest intents that are not in the corpus, which can have detrimental effects in performance. To address this, we propose dataset augmentation methods that align the ground truth clarifications with the retrieval corpus. Additionally, we explore techniques to enhance the relevance of the evidence pool during inference, but find that identifying ground truth intents within the corpus remains challenging. Our analysis suggests that this challenge is partly due to the bias of current datasets towards clarification taxonomies and calls for data that can support generating corpus-informed clarifications.

Related papers

• Likelihood as a Performance Gauge for Retrieval-Augmented Generation [78.28197013467157]
  We show that likelihoods serve as an effective gauge for language model performance. We propose two methods that use question likelihood as a gauge for selecting and constructing prompts that lead to better performance.
  arXiv  Detail & Related papers  (2024-11-12T13:14:09Z)
• Corpus-Steered Query Expansion with Large Language Models [35.64662397095323]
  We introduce Corpus-Steered Query Expansion (CSQE) to promote the incorporation of knowledge embedded within the corpus. CSQE utilizes the relevance assessing capability of LLMs to systematically identify pivotal sentences in the initially-retrieved documents. Extensive experiments reveal that CSQE exhibits strong performance without necessitating any training, especially with queries for which LLMs lack knowledge.
  arXiv  Detail & Related papers  (2024-02-28T03:58:58Z)
• PAQA: Toward ProActive Open-Retrieval Question Answering [34.883834970415734]
  This work aims to tackle the challenge of generating relevant clarifying questions by taking into account the inherent ambiguities present in both user queries and documents. We propose PAQA, an extension to the existing AmbiNQ dataset, incorporating clarifying questions. We then evaluate various models and assess how passage retrieval impacts ambiguity detection and the generation of clarifying questions.
  arXiv  Detail & Related papers  (2024-02-26T14:40:34Z)
• Qsnail: A Questionnaire Dataset for Sequential Question Generation [76.616068047362]
  We present the first dataset specifically constructed for the questionnaire generation task, which comprises 13,168 human-written questionnaires. We conduct experiments on Qsnail, and the results reveal that retrieval models and traditional generative models do not fully align with the given research topic and intents. Despite enhancements through the chain-of-thought prompt and finetuning, questionnaires generated by language models still fall short of human-written questionnaires.
  arXiv  Detail & Related papers  (2024-02-22T04:14:10Z)
• Towards Reliable and Factual Response Generation: Detecting Unanswerable Questions in Information-Seeking Conversations [16.99952884041096]
  Generative AI models face the challenge of hallucinations that can undermine users' trust in such systems. We approach the problem of conversational information seeking as a two-step process, where relevant passages in a corpus are identified first and then summarized into a final system response. Specifically, our proposed method employs a sentence-level classifier to detect if the answer is present, then aggregates these predictions on the passage level, and eventually across the top-ranked passages to arrive at a final answerability estimate.
  arXiv  Detail & Related papers  (2024-01-21T10:15:36Z)
• DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text [73.68051228972024]
  Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when relying on their internal knowledge. Retrieval-augmented LLMs have emerged as a potential solution to ground LLMs in external knowledge.
  arXiv  Detail & Related papers  (2023-10-31T04:37:57Z)
• Enhancing Argument Structure Extraction with Efficient Leverage of Contextual Information [79.06082391992545]
  We propose an Efficient Context-aware model (ECASE) that fully exploits contextual information. We introduce a sequence-attention module and distance-weighted similarity loss to aggregate contextual information and argumentative information. Our experiments on five datasets from various domains demonstrate that our model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-10-08T08:47:10Z)
• Inducing Causal Structure for Abstractive Text Summarization [76.1000380429553]
  We introduce a Structural Causal Model (SCM) to induce the underlying causal structure of the summarization data. We propose a Causality Inspired Sequence-to-Sequence model (CI-Seq2Seq) to learn the causal representations that can mimic the causal factors. Experimental results on two widely used text summarization datasets demonstrate the advantages of our approach.
  arXiv  Detail & Related papers  (2023-08-24T16:06:36Z)
• Adapting the NICT-JLE Corpus for Disfluency Detection Models [9.90780328490921]
  This paper describes the adaptation of the NICT-JLE corpus to a format suitable for disfluency detection model training and evaluation. Points of difference between the NICT-JLE and Switchboard corpora are explored, followed by a detailed overview of adaptations to the tag set and meta-features. The result of this work provides a standardised train, heldout and test set for use in future research on disfluency detection for learner speech.
  arXiv  Detail & Related papers  (2023-08-04T17:54:52Z)
• Evaluating Factuality in Generation with Dependency-level Entailment [57.5316011554622]
  We propose a new formulation of entailment that decomposes it at the level of dependency arcs. We show that our dependency arc entailment model trained on this data can identify factual inconsistencies in paraphrasing and summarization better than sentence-level methods.
  arXiv  Detail & Related papers  (2020-10-12T06:43:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.