Failure Modes in Multi-Hop QA: The Weakest Link Law and the Recognition Bottleneck

• URL: http://arxiv.org/abs/2601.12499v1
• Date: Sun, 18 Jan 2026 17:16:04 GMT
• Title: Failure Modes in Multi-Hop QA: The Weakest Link Law and the Recognition Bottleneck
• Abstract summary: We introduce Multi-Focus Attention Instruction (MFAI) to disentangle mechanisms by explicitly steering attention towards selected positions.<n>We demonstrate that "thinking" models that utilize System-2 reasoning, effectively locate and integrate the required information.
• Score: 38.430205063059724
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite scaling to massive context windows, Large Language Models (LLMs) struggle with multi-hop reasoning due to inherent position bias, which causes them to overlook information at certain positions. Whether these failures stem from an inability to locate evidence (recognition failure) or integrate it (synthesis failure) is unclear. We introduce Multi-Focus Attention Instruction (MFAI), a semantic probe to disentangle these mechanisms by explicitly steering attention towards selected positions. Across 5 LLMs on two multi-hop QA tasks (MuSiQue and NeoQA), we establish the "Weakest Link Law": multi-hop reasoning performance collapses to the performance level of the least visible evidence. Crucially, this failure is governed by absolute position rather than the linear distance between facts (performance variance $<3%$). We further identify a duality in attention steering: while matched MFAI resolves recognition bottlenecks, improving accuracy by up to 11.5% in low-visibility positions, misleading MFAI triggers confusion in real-world tasks but is successfully filtered in synthetic tasks. Finally, we demonstrate that "thinking" models that utilize System-2 reasoning, effectively locate and integrate the required information, matching gold-only baselines even in noisy, long-context settings.

Related papers

• CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps [19.949184312340556]
  We introduce CRiT-QA (Counterfactual Reasoning with Traps), a dataset explicitly designed to address both limitations.<n>To neutralize reliance on memorized knowledge and enforce strict context dependency, CRiT-QA transforms factual reasoning chains with counterfactual entities.<n>Our experiments show that LLMs exhibit substantial performance degradation on CRiT-QA compared to standard datasets.
  arXiv  Detail & Related papers  (2026-07-12T04:40:02Z)
• METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models [61.33372454250959]
  Contextual causal reasoning is a critical yet challenging capability for Large Language Models.<n>Existing benchmarks fail to ensure context consistency or cover the full causal hierarchy.<n>We pioneer METER to systematically benchmark LLMs across all three levels of the causal ladder.
  arXiv  Detail & Related papers  (2026-04-13T14:07:11Z)
• HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help? [32.54022440678003]
  coding agents solve complex tasks when given complete context but collapse when specifications are incomplete or ambiguous.<n>Current benchmarks are blind to this failure mode.<n>We present HiL-Bench to measure this selective escalation skill.
  arXiv  Detail & Related papers  (2026-04-10T15:21:44Z)
• Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models [60.418191092851636]
  Omanic is an open-domain multi-hop QA resource that provides decomposed sub-questions and intermediate answers as structural annotations for analyzing reasoning processes.<n>It contains 10,296 machine-generated training examples (Omanic Synth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench)
  arXiv  Detail & Related papers  (2026-03-17T15:23:37Z)
• Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning [16.274791437311602]
  We introduce ARMOR: Adaptive Round-based Multi-task mOdel for Robotic failure detection and reasoning.<n>We formulate detection and reasoning as a multi-task self-refinement process.<n>We show that ARMOR achieves state-of-the-art performance by improving over the previous approaches by up to 30% on failure detection rate.
  arXiv  Detail & Related papers  (2026-02-12T20:55:36Z)
• Hallucination Detection via Internal States and Structured Reasoning Consistency in Large Language Models [7.18947815679122]
  Internal State Probing and Chain-of-Thought Verification are used to detect hallucinations in large language models.<n>We develop a unified framework that bridges the gap between the two methods.<n>Our framework consistently and significantly outperforms strong baselines.
  arXiv  Detail & Related papers  (2025-10-13T15:31:21Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning [3.949628618389608]
  AURA is a benchmark for evaluating the cross-modal reasoning capabilities of Audio-Visual Large Language Models (AV-LLMs) and Omni-modal Language Models (OLMs)<n>AURA includes questions across six challenging cognitive domains, such as causality, timbre and pitch, tempo and AV synchronization, unanswerability, implicit distractions, and skill profiling.<n>We propose a novel metric, AuraScore, which addresses the lack of robust tools for evaluating reasoning fidelity.
  arXiv  Detail & Related papers  (2025-08-10T20:06:42Z)
• Reinforcing Video Reasoning with Focused Thinking [65.85683941058916]
  We propose TW-GRPO, a novel framework that enhances visual reasoning with focused thinking and dense reward granularity.<n>Specifically, we employ a token weighting mechanism that prioritizes tokens with high informational density.<n>We also reformulate RL training by shifting from single-choice to multi-choice QA tasks.
  arXiv  Detail & Related papers  (2025-05-30T15:42:19Z)
• Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation [56.69064935192318]
  Multi-hop Question Answering (MHQA) adds layers of complexity to question answering, making it more challenging.<n>This paper explores how Language Models respond to multi-hop questions by permuting search results (retrieved documents) under various configurations.
  arXiv  Detail & Related papers  (2025-05-16T23:29:47Z)
• SG-FSM: A Self-Guiding Zero-Shot Prompting Paradigm for Multi-Hop Question Answering Based on Finite State Machine [27.274219226254026]
  Multi-hop Question Answering (MHQA) remains challenging for many existing models. We propose the Self-Guiding prompting Finite State Machine (SG-FSM) to strengthen multi-hop reasoning abilities.
  arXiv  Detail & Related papers  (2024-10-22T13:47:38Z)
• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs)<n>Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings.<n>By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making.<n>We present a process-based benchmark MR-Ben that demands a meta-reasoning skill.<n>Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.