NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts

• URL: http://arxiv.org/abs/2506.02000v2
• Date: Mon, 23 Jun 2025 01:41:05 GMT
• Title: NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts
• Abstract summary: NovelHopQA is the first benchmark to evaluate 1-4 hop QA over 64k-128k-separated excerpts from 83 full-length novels.<n>A keyword-guided pipeline builds hop-token chains grounded in coherent storylines.
• Score: 4.716662397532276
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current large language models (LLMs) struggle to answer questions that span tens of thousands of tokens, especially when multi-hop reasoning is involved. While prior benchmarks explore long-context comprehension or multi-hop reasoning in isolation, none jointly vary context length and reasoning depth in natural narrative settings. We introduce NovelHopQA, the first benchmark to evaluate 1-4 hop QA over 64k-128k-token excerpts from 83 full-length public-domain novels. A keyword-guided pipeline builds hop-separated chains grounded in coherent storylines. We evaluate seven state-of-the-art models and apply oracle-context filtering to ensure all questions are genuinely answerable. Human annotators validate both alignment and hop depth. We additionally present retrieval-augmented generation (RAG) evaluations to test model performance when only selected passages are provided instead of the full context. We noticed consistent accuracy drops with increased hops and context length increase, even for frontier models-revealing that sheer scale does not guarantee robust reasoning. Failure-mode analysis highlights common breakdowns such as missed final-hop integration and long-range drift. NovelHopQA offers a controlled diagnostic setting to test multi-hop reasoning at scale. All code and datasets are available at https://novelhopqa.github.io.

Related papers

• HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning [86.82637240330791]
  Long chain-of-thought (CoT) reasoning exposes diverse failure modes.<n>Most existing vision-language data used for reinforcement learning with verifiable rewards does not involve complex reasoning chains.<n>We propose HopChain, a scalable framework for synthesizing multi-hop vision-language reasoning data.
  arXiv  Detail & Related papers  (2026-03-17T18:04:58Z)
• Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Models [26.603700269575025]
  We show that bridge entities are computed sequentially across layers before later-hop answers.<n>We propose a framework that models multi-hop reasoning as broad recall in shallow layers followed by selective extraction in deeper attention layers.
  arXiv  Detail & Related papers  (2026-01-07T03:13:03Z)
• PRISM: Agentic Retrieval with LLMs for Multi-Hop Question Answering [10.971852280240357]
  We introduce an Agentic Retrieval System that leverages large language models (LLMs) in a structured loop to retrieve relevant evidence with high precision and recall.<n>Our framework consists of three specialized agents: a Question Analyzer that decomposes a multi-hop question into sub-questions, a Selector that identifies the most relevant context for each sub-question, and an Adder that brings in any missing evidence.
  arXiv  Detail & Related papers  (2025-10-16T04:02:29Z)
• Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation [56.69064935192318]
  Multi-hop Question Answering (MHQA) adds layers of complexity to question answering, making it more challenging.<n>This paper explores how Language Models respond to multi-hop questions by permuting search results (retrieved documents) under various configurations.
  arXiv  Detail & Related papers  (2025-05-16T23:29:47Z)
• SCoRE: Benchmarking Long-Chain Reasoning in Commonsense Scenarios [33.72114830484246]
  We introduce SCoRE (Scenario-based Commonsense Reasoning Evaluation), a benchmark that synthesizes multi-hop questions from scenario schemas of entities, relations, and logical rules.<n>SCoRE contains 100k bilingual (Chinese-English) multiple-choice questions whose reasoning chains span 2-11 hops and are grouped into various difficulty levels.
  arXiv  Detail & Related papers  (2025-03-08T13:40:10Z)
• End-to-End Beam Retrieval for Multi-Hop Question Answering [37.13580394608824]
  Multi-hop question answering involves finding multiple relevant passages and step-by-step reasoning to answer complex questions. Previous retrievers were customized for two-hop questions, and most of them were trained separately across different hops. We introduce Beam Retrieval, an end-to-end beam retrieval framework for multi-hop QA.
  arXiv  Detail & Related papers  (2023-08-17T13:24:14Z)
• Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering [85.79940770146557]
  We decompose multi-hop questions into multiple corresponding single-hop questions. We find marked inconsistency in QA models' answers on these pairs of ostensibly identical question chains. When trained only on single-hop questions, models generalize poorly to multi-hop questions.
  arXiv  Detail & Related papers  (2022-10-09T11:48:07Z)
• Locate Then Ask: Interpretable Stepwise Reasoning for Multi-hop Question Answering [71.49131159045811]
  Multi-hop reasoning requires aggregating multiple documents to answer a complex question. Existing methods usually decompose the multi-hop question into simpler single-hop questions. We propose an interpretable stepwise reasoning framework to incorporate both single-hop supporting sentence identification and single-hop question generation.
  arXiv  Detail & Related papers  (2022-08-22T13:24:25Z)
• Teaching Broad Reasoning Skills via Decomposition-Guided Contexts [50.114651561111245]
  Question-answering datasets require a broad set of reasoning skills. We show how to use question decompositions to teach these broad reasoning skills in a robust fashion.
  arXiv  Detail & Related papers  (2022-05-25T05:13:21Z)
• Modeling Multi-hop Question Answering as Single Sequence Prediction [88.72621430714985]
  We propose a simple generative approach (PathFid) that extends the task beyond just answer generation. PathFid explicitly models the reasoning process to resolve the answer for multi-hop questions. Our experiments demonstrate that PathFid leads to strong performance gains on two multi-hop QA datasets.
  arXiv  Detail & Related papers  (2022-05-18T21:57:59Z)
• Reasoning Chain Based Adversarial Attack for Multi-hop Question Answering [0.0]
  Previous adversarial attack works usually edit the whole question sentence. We propose a multi-hop reasoning chain based adversarial attack method. Results demonstrate significant performance reduction on both answer and supporting facts prediction.
  arXiv  Detail & Related papers  (2021-12-17T18:03:14Z)
• Multi-hop Question Generation with Graph Convolutional Network [58.31752179830959]
  Multi-hop Question Generation (QG) aims to generate answer-related questions by aggregating and reasoning over multiple scattered evidence from different paragraphs. We propose Multi-Hop volution Fusion Network for Question Generation (MulQG), which does context encoding in multiple hops. Our proposed model is able to generate fluent questions with high completeness and outperforms the strongest baseline by 20.8% in the multi-hop evaluation.
  arXiv  Detail & Related papers  (2020-10-19T06:15:36Z)
• Counterfactual Variable Control for Robust and Interpretable Question Answering [57.25261576239862]
  Deep neural network based question answering (QA) models are neither robust nor explainable in many cases. In this paper, we inspect such spurious "capability" of QA models using causal inference. We propose a novel approach called Counterfactual Variable Control (CVC) that explicitly mitigates any shortcut correlation.
  arXiv  Detail & Related papers  (2020-10-12T10:09:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.