Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations

• URL: http://arxiv.org/abs/2608.00711v1
• Date: Sat, 01 Aug 2026 15:19:10 GMT
• Title: Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations
• Abstract summary: SCHEMA is an evidence-grounded, topology-aware evaluation framework for hallucinations in scientific agents.<n>We show that hallucinations concentrate at a small set of highly connected knowledge hubs.<n>Results indicate that for high-stakes scientific applications, terminal accuracy alone is an insufficient signal of agent reliability.
• Score: 27.330265411994258
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM) agents are increasingly deployed in scientific research, where reliability is critical and the underlying knowledge is densely interconnected. In such settings, hallucinations are particularly damaging: a single erroneous claim on a foundational concept can propagate through multi-step reasoning and corrupt entire trajectories. Existing hallucination benchmarks largely operate at the surface level, treating facts in isolation and relying on uniform accuracy metrics that ignore this topological structure. We address this gap with SCHEMA, the first evidence-grounded, topology-aware evaluation framework for hallucinations in scientific agents. SCHEMA automatically constructs scientific concept graphs from benchmark seeds and literature evidence, synthesizes graph-grounded tasks spanning claim verification, multi-hop reasoning, open-ended explanation, and experimental code generation, and evaluates agents with two complementary diagnostics. A trajectory hallucination pipeline audits intermediate reasoning at scale via a topology-weighted severity score, while a multi-agent counterfactual attribution module pinpoints the causal mechanism behind selected failures. SCHEMA reveals that hallucinations concentrate at a small set of highly connected knowledge hubs, and that final-answer accuracy decouples from trajectory honesty; models often reach correct conclusions through structurally flawed reasoning. These results indicate that for high-stakes scientific applications, terminal accuracy alone is an insufficient signal of agent reliability, motivating mechanism-level evaluation grounded in knowledge topology. Code is available at https://github.com/circles-post/SCHEMA.

Related papers

• Agentic Root Cause Analysis through Evidence-Grounded Reasoning [16.37572484400141]
  AgentRCA is a zero-shot agentic framework for evidence-grounded root cause analysis.<n>It performs inference-time reasoning by combining a data-driven digital twin with a tool-augmented large language model.<n>It achieves diagnostic performance competitive with fully supervised baselines without relying on fault-specific training.
  arXiv  Detail & Related papers  (2026-07-24T15:10:41Z)
• Accurate, Interdisciplinary and Transparent Structure-property Understanding with Deep Native Structural Reasoning [103.788281428856]
  We introduce SciReasoner, a scientific foundation model for native structural reasoning across proteins, small molecules and inorganic crystals.<n>SciReasoner discretizes coordinates, topologies and periodic connectivities into a unified structure-aware vocabulary.<n>Across 86 benchmarks, SciReasoner achieves state-of-the-art performance on 67 tasks.
  arXiv  Detail & Related papers  (2026-07-08T17:59:59Z)
• HypoAgent: An Agentic Framework for Interactive Abductive Hypothesis Generation over Knowledge Graphs [52.91976391891331]
  Abductive reasoning over knowledge graphs aims to generate logical hypotheses that explain observed entities or facts.<n>Existing controllable hypothesis generation methods allow users to guide this process with explicit conditions.<n>We propose HypoAgent, an Agentic framework for interactive abductive Hypothesis Generation over knowledge graphs.
  arXiv  Detail & Related papers  (2026-05-29T14:40:37Z)
• Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models [62.932580559941414]
  Vision-Language Models (VLMs) frequently "hallucinate" - generate plausible yet factually incorrect statements.<n>We propose a new paradigm for diagnosing hallucinations, recasting them from static output errors into dynamic pathologies of a model's computational cognition.
  arXiv  Detail & Related papers  (2026-03-16T17:20:38Z)
• Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectory [7.41972830438376]
  Diagnosing the failure mechanisms of Deep Research Agents (DRAs) remains a critical challenge.<n>Existing benchmarks predominantly rely on end-to-end evaluation, obscuring critical intermediate hallucinations.<n>We introduce the PIES taxonomy to categorize hallucinations along functional components and error properties.
  arXiv  Detail & Related papers  (2026-01-30T13:49:09Z)
• Atomic Reasoning for Scientific Table Claim Verification [83.14588611859826]
  Non-experts are susceptible to misleading claims based on scientific tables due to their high information density and perceived credibility.<n>Existing table claim verification models, including state-of-the-art large language models (LLMs), often struggle with precise fine-grained reasoning.<n>Inspired by Cognitive Load Theory, we propose that enhancing a model's ability to interpret table-based claims involves reducing cognitive load.
  arXiv  Detail & Related papers  (2025-06-08T02:46:22Z)
• Joint Evaluation of Answer and Reasoning Consistency for Hallucination Detection in Large Reasoning Models [12.270274049887298]
  Reasoning traces can be redundant or logically inconsistent, making them a new source of hallucination.<n>Existing hallucination detection methods focus primarily on answer-level uncertainty.<n>We propose RACE, a novel framework specifically tailored for hallucination detection in LRMs.
  arXiv  Detail & Related papers  (2025-06-05T09:54:04Z)
• Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective [11.013059864022667]
  Reasoning Hallucinations are logically coherent but factually incorrect reasoning traces.<n>These errors are embedded within structured reasoning, making them more difficult to detect and potentially more harmful.<n>We propose the Reasoning Score, which quantifies the depth of reasoning by measuring the divergence between logits.<n>We also introduce GRPO-R, an enhanced reinforcement learning algorithm that incorporates step-level deep reasoning rewards via potential-based shaping.
  arXiv  Detail & Related papers  (2025-05-19T09:16:40Z)
• Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations [82.42811602081692]
  This paper introduces a subsequence association framework to systematically trace and understand hallucinations.<n>Key insight is hallucinations that arise when dominant hallucinatory associations outweigh faithful ones.<n>We propose a tracing algorithm that identifies causal subsequences by analyzing hallucination probabilities across randomized input contexts.
  arXiv  Detail & Related papers  (2025-04-17T06:34:45Z)
• Context-Aware Reasoning On Parametric Knowledge for Inferring Causal Variables [49.31233968546582]
  We introduce a novel benchmark where the objective is to complete a partial causal graph.<n>We show the strong ability of LLMs to hypothesize the backdoor variables between a cause and its effect.<n>Unlike simple memorization of fixed associations, our task requires the LLM to reason according to the context of the entire graph.
  arXiv  Detail & Related papers  (2024-09-04T10:37:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.