Dr3: Ask Large Language Models Not to Give Off-Topic Answers in Open Domain Multi-Hop Question Answering

• URL: http://arxiv.org/abs/2403.12393v1
• Date: Tue, 19 Mar 2024 03:00:03 GMT
• Title: Dr3: Ask Large Language Models Not to Give Off-Topic Answers in Open Domain Multi-Hop Question Answering
• Abstract summary: Open Domain Multi-Hop Question Answering (ODMHQA) plays a crucial role in Natural Language Processing (NLP) Large Language Models (LLMs) have demonstrated remarkable performance in solving ODMHQA. LLMs may generate off-topic answers when attempting to solve ODMHQA.
• Score: 27.72516535251282
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open Domain Multi-Hop Question Answering (ODMHQA) plays a crucial role in Natural Language Processing (NLP) by aiming to answer complex questions through multi-step reasoning over retrieved information from external knowledge sources. Recently, Large Language Models (LLMs) have demonstrated remarkable performance in solving ODMHQA owing to their capabilities including planning, reasoning, and utilizing tools. However, LLMs may generate off-topic answers when attempting to solve ODMHQA, namely the generated answers are irrelevant to the original questions. This issue of off-topic answers accounts for approximately one-third of incorrect answers, yet remains underexplored despite its significance. To alleviate this issue, we propose the Discriminate->Re-Compose->Re- Solve->Re-Decompose (Dr3) mechanism. Specifically, the Discriminator leverages the intrinsic capabilities of LLMs to judge whether the generated answers are off-topic. In cases where an off-topic answer is detected, the Corrector performs step-wise revisions along the reversed reasoning chain (Re-Compose->Re-Solve->Re-Decompose) until the final answer becomes on-topic. Experimental results on the HotpotQA and 2WikiMultiHopQA datasets demonstrate that our Dr3 mechanism considerably reduces the occurrence of off-topic answers in ODMHQA by nearly 13%, improving the performance in Exact Match (EM) by nearly 3% compared to the baseline method without the Dr3 mechanism.

Related papers

• Inferential Question Answering [67.54465021408724]
  We introduce Inferential QA -- a new task that challenges models to infer answers from answer-supporting passages which provide only clues.<n>To study this problem, we construct QUIT (QUestions requiring Inference from Texts) dataset, comprising 7,401 questions and 2.4M passages.<n>We show that methods effective on traditional QA tasks struggle in inferential QA: retrievers underperform, rerankers offer limited gains, and fine-tuning provides inconsistent improvements.
  arXiv  Detail & Related papers  (2026-02-01T14:02:43Z)
• Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation [56.69064935192318]
  Multi-hop Question Answering (MHQA) adds layers of complexity to question answering, making it more challenging.<n>This paper explores how Language Models respond to multi-hop questions by permuting search results (retrieved documents) under various configurations.
  arXiv  Detail & Related papers  (2025-05-16T23:29:47Z)
• Mitigating Lost-in-Retrieval Problems in Retrieval Augmented Multi-Hop Question Answering [25.544345244474073]
  We identify a critical problem, "lost-in-retrieval", in in-augmented multi-hop question answering (QA) The key entities are missed in subquestion decomposition. We propose three retrieval chain models: GP4o-mini, Qwen-2.5-72 and GLM-4-2.5-72. Empirical results demonstrate that GP4o-mini consistently outperforms baselines in both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2025-02-20T04:19:05Z)
• Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent [102.31558123570437]
  Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs) We propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch.
  arXiv  Detail & Related papers  (2024-11-05T09:27:21Z)
• GenSco: Can Question Decomposition based Passage Alignment improve Question Answering? [1.5776201492893507]
  "GenSco" is a novel approach of selecting passages based on the predicted decomposition of the multi-hop questions. We evaluate on three broadly established multi-hop question answering datasets.
  arXiv  Detail & Related papers  (2024-07-14T15:25:08Z)
• GenDec: A robust generative Question-decomposition method for Multi-hop reasoning [32.12904215053187]
  Multi-hop QA involves step-by-step reasoning to answer complex questions. Existing large language models'(LLMs) reasoning ability in multi-hop question answering remains exploration. It is unclear whether LLMs follow a desired reasoning chain to reach the right final answer.
  arXiv  Detail & Related papers  (2024-02-17T02:21:44Z)
• Answering Questions by Meta-Reasoning over Multiple Chains of Thought [53.55653437903948]
  We introduce Multi-Chain Reasoning (MCR), an approach which prompts large language models to meta-reason over multiple chains of thought. MCR examines different reasoning chains, mixes information between them and selects the most relevant facts in generating an explanation and predicting the answer.
  arXiv  Detail & Related papers  (2023-04-25T17:27:37Z)
• Successive Prompting for Decomposing Complex Questions [50.00659445976735]
  Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting. We introduce Successive Prompting'', where we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution. Our best model (with successive prompting) achieves an improvement of 5% absolute F1 on a few-shot version of the DROP dataset.
  arXiv  Detail & Related papers  (2022-12-08T06:03:38Z)
• Interpretable AMR-Based Question Decomposition for Multi-hop Question Answering [12.35571328854374]
  We propose a Question Decomposition method based on Abstract Meaning Representation (QDAMR) for multi-hop QA. We decompose a multi-hop question into simpler sub-questions and answer them in order. Experimental results on HotpotQA demonstrate that our approach is competitive for interpretable reasoning.
  arXiv  Detail & Related papers  (2022-06-16T23:46:33Z)
• Multifaceted Improvements for Conversational Open-Domain Question Answering [54.913313912927045]
  We propose a framework with Multifaceted Improvements for Conversational open-domain Question Answering (MICQA) Firstly, the proposed KL-divergence based regularization is able to lead to a better question understanding for retrieval and answer reading. Second, the added post-ranker module can push more relevant passages to the top placements and be selected for reader with a two-aspect constrains. Third, the well designed curriculum learning strategy effectively narrows the gap between the golden passage settings of training and inference, and encourages the reader to find true answer without the golden passage assistance.
  arXiv  Detail & Related papers  (2022-04-01T07:54:27Z)
• Unsupervised Question Decomposition for Question Answering [102.56966847404287]
  We propose an algorithm for One-to-N Unsupervised Sequence Sequence (ONUS) that learns to map one hard, multi-hop question to many simpler, single-hop sub-questions. We show large QA improvements on HotpotQA over a strong baseline on the original, out-of-domain, and multi-hop dev sets.
  arXiv  Detail & Related papers  (2020-02-22T19:40:35Z)
• Break It Down: A Question Understanding Benchmark [79.41678884521801]
  We introduce a Question Decomposition Representation Meaning (QDMR) for questions. QDMR constitutes the ordered list of steps, expressed through natural language, that are necessary for answering a question. We release the Break dataset, containing over 83K pairs of questions and their QDMRs.
  arXiv  Detail & Related papers  (2020-01-31T11:04:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.