DEEPAMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness

• URL: http://arxiv.org/abs/2511.01323v1
• Date: Mon, 03 Nov 2025 08:15:24 GMT
• Title: DEEPAMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness
• Abstract summary: We build a dataset of 3,600 questions requiring multi-hop reasoning and half of them explicit name ambiguity resolving.<n>Experiments reveal that, even state-of-the-art GPT-5 show incomplete answers, achieving only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous questions.
• Score: 28.51983430422024
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) with integrated search tools show strong promise in open-domain question answering (QA), yet they often struggle to produce complete answer set to complex questions such as Which actor from the film Heat won at least one Academy Award?, which requires (1) distinguishing between multiple films sharing the same title and (2) reasoning across a large set of actors to gather and integrate evidence. Existing QA benchmarks rarely evaluate both challenges jointly. To address this, we introduce DeepAmbigQAGen, an automatic data generation pipeline that constructs QA tasks grounded in text corpora and linked knowledge graph, generating natural and verifiable questions that systematically embed name ambiguity and multi-step reasoning. Based on this, we build DeepAmbigQA, a dataset of 3,600 questions requiring multi-hop reasoning and half of them explicit name ambiguity resolving. Experiments reveal that, even state-of-the-art GPT-5 show incomplete answers, achieving only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous questions. These findings highlight the need for more robust QA systems aimed at information gathering and answer completeness.

Related papers

• Exploiting Intrinsic Duality for Multi-Hop Question Generation [9.34607982361678]
  Multi hop question generation (MQG) aims to generate questions from multiple given documents and target answers, whereas question answering (QA) focuses on deriving answers from documents given specific questions.<n>We propose QQ, a novel framework that exploits the duality between Question and answer for multi hop Question generation.<n>QQ employs a unified architecture functioning simultaneously as both an MQG and a QA model to fully leverage their interdependence.
  arXiv  Detail & Related papers  (2026-08-01T15:20:16Z)
• Inferential Question Answering [67.54465021408724]
  We introduce Inferential QA -- a new task that challenges models to infer answers from answer-supporting passages which provide only clues.<n>To study this problem, we construct QUIT (QUestions requiring Inference from Texts) dataset, comprising 7,401 questions and 2.4M passages.<n>We show that methods effective on traditional QA tasks struggle in inferential QA: retrievers underperform, rerankers offer limited gains, and fine-tuning provides inconsistent improvements.
  arXiv  Detail & Related papers  (2026-02-01T14:02:43Z)
• PRISM: Agentic Retrieval with LLMs for Multi-Hop Question Answering [10.971852280240357]
  We introduce an Agentic Retrieval System that leverages large language models (LLMs) in a structured loop to retrieve relevant evidence with high precision and recall.<n>Our framework consists of three specialized agents: a Question Analyzer that decomposes a multi-hop question into sub-questions, a Selector that identifies the most relevant context for each sub-question, and an Adder that brings in any missing evidence.
  arXiv  Detail & Related papers  (2025-10-16T04:02:29Z)
• MIRAGE: Multi-hop Reasoning with Ambiguity Evaluation for Illusory Questions [25.695038634265]
  Real-world Multi-hop Question Answering (QA) often involves ambiguity that is inseparable from the reasoning process itself.<n>This ambiguity creates a distinct challenge, where multiple reasoning paths emerge from a single question.<n>We introduce MultI-hop Reasoning with AmbiGuity Evaluation for Illusory Questions (MIRAGE) to analyze and evaluate this challenging intersection.
  arXiv  Detail & Related papers  (2025-09-26T07:31:01Z)
• Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations [3.3018718917393297]
  We propose the novel task of Question Answering with source citation in ambiguous settings, where multiple valid answers exist. We create a comprehensive framework consisting of: (1) five novel datasets; (2) the first ambiguous multi-hop QA dataset featuring real-world, naturally occurring contexts; and (3) two new metrics to evaluate models' performances. We hope that this new task, datasets, metrics, and baselines will inspire the community to push the boundaries of QA research and develop more trustworthy and interpretable systems.
  arXiv  Detail & Related papers  (2024-10-05T17:37:01Z)
• Multi-LLM QA with Embodied Exploration [55.581423861790945]
  We investigate the use of Multi-Embodied LLM Explorers (MELE) for question-answering in an unknown environment. Multiple LLM-based agents independently explore and then answer queries about a household environment. We analyze different aggregation methods to generate a single, final answer for each query.
  arXiv  Detail & Related papers  (2024-06-16T12:46:40Z)
• GenDec: A robust generative Question-decomposition method for Multi-hop reasoning [32.12904215053187]
  Multi-hop QA involves step-by-step reasoning to answer complex questions. Existing large language models'(LLMs) reasoning ability in multi-hop question answering remains exploration. It is unclear whether LLMs follow a desired reasoning chain to reach the right final answer.
  arXiv  Detail & Related papers  (2024-02-17T02:21:44Z)
• Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models [30.186503757127188]
  Tree of Clarifications (ToC) is a framework to generate a long-form answer to ambiguous questions. ToC outperforms existing baselines on ASQA in a few-shot setup across the metrics.
  arXiv  Detail & Related papers  (2023-10-23T08:42:49Z)
• Modeling Multi-hop Question Answering as Single Sequence Prediction [88.72621430714985]
  We propose a simple generative approach (PathFid) that extends the task beyond just answer generation. PathFid explicitly models the reasoning process to resolve the answer for multi-hop questions. Our experiments demonstrate that PathFid leads to strong performance gains on two multi-hop QA datasets.
  arXiv  Detail & Related papers  (2022-05-18T21:57:59Z)
• Multifaceted Improvements for Conversational Open-Domain Question Answering [54.913313912927045]
  We propose a framework with Multifaceted Improvements for Conversational open-domain Question Answering (MICQA) Firstly, the proposed KL-divergence based regularization is able to lead to a better question understanding for retrieval and answer reading. Second, the added post-ranker module can push more relevant passages to the top placements and be selected for reader with a two-aspect constrains. Third, the well designed curriculum learning strategy effectively narrows the gap between the golden passage settings of training and inference, and encourages the reader to find true answer without the golden passage assistance.
  arXiv  Detail & Related papers  (2022-04-01T07:54:27Z)
• ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers [93.55268936974971]
  We describe a Question Answering dataset that contains complex questions with conditional answers. We call this dataset ConditionalQA. We show that ConditionalQA is challenging for many of the existing QA models, especially in selecting answer conditions.
  arXiv  Detail & Related papers  (2021-10-13T17:16:46Z)
• Multi-hop Question Generation with Graph Convolutional Network [58.31752179830959]
  Multi-hop Question Generation (QG) aims to generate answer-related questions by aggregating and reasoning over multiple scattered evidence from different paragraphs. We propose Multi-Hop volution Fusion Network for Question Generation (MulQG), which does context encoding in multiple hops. Our proposed model is able to generate fluent questions with high completeness and outperforms the strongest baseline by 20.8% in the multi-hop evaluation.
  arXiv  Detail & Related papers  (2020-10-19T06:15:36Z)
• Unsupervised Question Decomposition for Question Answering [102.56966847404287]
  We propose an algorithm for One-to-N Unsupervised Sequence Sequence (ONUS) that learns to map one hard, multi-hop question to many simpler, single-hop sub-questions. We show large QA improvements on HotpotQA over a strong baseline on the original, out-of-domain, and multi-hop dev sets.
  arXiv  Detail & Related papers  (2020-02-22T19:40:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.