Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA

• URL: http://arxiv.org/abs/2409.15515v1
• Date: Mon, 23 Sep 2024 20:05:12 GMT
• Title: Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA
• Authors: Nirmal Roy, Leonardo F. R. Ribeiro, Rexhina Blloshmi, Kevin Small,
• Abstract summary: We build on the single-turn SELF-RAG framework and propose SELF-multi-RAG for conversational settings. SELF-multi-RAG demonstrates improved capabilities over single-turn variants with respect to retrieving relevant passages.
• Score: 16.1357049130957
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Augmenting Large Language Models (LLMs) with information retrieval capabilities (i.e., Retrieval-Augmented Generation (RAG)) has proven beneficial for knowledge-intensive tasks. However, understanding users' contextual search intent when generating responses is an understudied topic for conversational question answering (QA). This conversational extension leads to additional concerns when compared to single-turn QA as it is more challenging for systems to comprehend conversational context and manage retrieved passages over multiple turns. In this work, we propose a method for enabling LLMs to decide when to retrieve in RAG settings given a conversational context. When retrieval is deemed necessary, the LLM then rewrites the conversation for passage retrieval and judges the relevance of returned passages before response generation. Operationally, we build on the single-turn SELF-RAG framework (Asai et al., 2023) and propose SELF-multi-RAG for conversational settings. SELF-multi-RAG demonstrates improved capabilities over single-turn variants with respect to retrieving relevant passages (by using summarized conversational context) and assessing the quality of generated responses. Experiments on three conversational QA datasets validate the enhanced response generation capabilities of SELF-multi-RAG, with improvements of ~13% measured by human annotation.

Related papers

• RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues [8.036117602566074]
  RAD-Bench is a benchmark designed to evaluate Large Language Models' capabilities in multi-turn dialogues following retrievals. Our evaluation results on commonly used LLMs reveal that model performance deteriorates as additional layers of conditions or constraints are applied.
  arXiv  Detail & Related papers  (2024-09-19T08:26:45Z)
• Boosting Conversational Question Answering with Fine-Grained Retrieval-Augmentation and Self-Check [25.63538452425097]
  We propose a conversation-level RAG approach, which incorporates fine-grained retrieval augmentation and self-check for conversational question answering. In particular, our approach consists of three components, namely conversational question refiner, fine-grained retriever and self-check based response generator.
  arXiv  Detail & Related papers  (2024-03-27T04:20:18Z)
• Effective and Efficient Conversation Retrieval for Dialogue State Tracking with Implicit Text Summaries [48.243879779374836]
  Few-shot dialogue state tracking (DST) with Large Language Models (LLM) relies on an effective and efficient conversation retriever to find similar in-context examples for prompt learning. Previous works use raw dialogue context as search keys and queries, and a retriever is fine-tuned with annotated dialogues to achieve superior performance. We handle the task of conversation retrieval based on text summaries of the conversations. A LLM-based conversation summarizer is adopted for query and key generation, which enables effective maximum inner product search.
  arXiv  Detail & Related papers  (2024-02-20T14:31:17Z)
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection [74.51523859064802]
  We introduce a new framework called Self-Reflective Retrieval-Augmented Generation (Self-RAG) Self-RAG enhances an LM's quality and factuality through retrieval and self-reflection. It significantly outperforms state-of-the-art LLMs and retrieval-augmented models on a diverse set of tasks.
  arXiv  Detail & Related papers  (2023-10-17T18:18:32Z)
• CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning [16.470428531658232]
  We develop a query rewriting model CONQRR that rewrites a conversational question in context into a standalone question. We show that CONQRR achieves state-of-the-art results on a recent open-domain CQA dataset.
  arXiv  Detail & Related papers  (2021-12-16T01:40:30Z)
• Open-Retrieval Conversational Machine Reading [80.13988353794586]
  In conversational machine reading, systems need to interpret natural language rules, answer high-level questions, and ask follow-up clarification questions. Existing works assume the rule text is provided for each user question, which neglects the essential retrieval step in real scenarios. In this work, we propose and investigate an open-retrieval setting of conversational machine reading.
  arXiv  Detail & Related papers  (2021-02-17T08:55:01Z)
• Reasoning in Dialog: Improving Response Generation by Context Reading Comprehension [49.92173751203827]
  In multi-turn dialog, utterances do not always take the full form of sentences. We propose to improve the response generation performance by examining the model's ability to answer a reading comprehension question.
  arXiv  Detail & Related papers  (2020-12-14T10:58:01Z)
• Open-Retrieval Conversational Question Answering [62.11228261293487]
  We introduce an open-retrieval conversational question answering (ORConvQA) setting, where we learn to retrieve evidence from a large collection before extracting answers. We build an end-to-end system for ORConvQA, featuring a retriever, a reranker, and a reader that are all based on Transformers.
  arXiv  Detail & Related papers  (2020-05-22T19:39:50Z)
• Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting [56.268862325167575]
  We tackle conversational passage retrieval (ConvPR) with query reformulation integrated into a multi-stage ad-hoc IR system. We propose two conversational query reformulation (CQR) methods: (1) term importance estimation and (2) neural query rewriting. For the former, we expand conversational queries using important terms extracted from the conversational context with frequency-based signals. For the latter, we reformulate conversational queries into natural, standalone, human-understandable queries with a pretrained sequence-tosequence model.
  arXiv  Detail & Related papers  (2020-05-05T14:30:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.