ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams

• URL: http://arxiv.org/abs/2604.15994v2
• Date: Thu, 23 Apr 2026 15:57:11 GMT
• Title: ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams
• Abstract summary: Multimodal Large Language Models (MLLMs) excel at recognizing individual visual elements and reasoning over simple linear diagrams.<n>We introduce ReactBench, a benchmark that reveals fundamental limitations in structural reasoning through chemical reaction diagrams.<n>Our benchmark comprises 1,618 expert-annotated QA pairs across four hierarchical task dimensions.
• Score: 21.265249070149842
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) excel at recognizing individual visual elements and reasoning over simple linear diagrams. However, when faced with complex topological structures involving branching paths, converging flows, and cyclic dependencies, their reasoning capabilities degrade sharply, even on tasks as basic as counting endpoints. Existing benchmarks fail to probe this gap, focusing on semantic comprehension rather than structural reasoning. We introduce ReactBench, a benchmark that reveals fundamental limitations in structural reasoning through chemical reaction diagrams. These real-world scientific diagrams offer an ideal testbed because they naturally span diverse structures from linear chains to cyclic graphs, while requiring both precise local recognition and coherent global reasoning. Our benchmark comprises 1,618 expert-annotated QA pairs across four hierarchical task dimensions. Extensive evaluation across 17 MLLMs reveals a significant performance gap exceeding 30% between anchor-based tasks and holistic structural reasoning tasks. Controlled ablations confirm this bottleneck lies in reasoning, not perception. These findings expose a fundamental deficit in structural understanding and establish directions for advancing visual reasoning.

Related papers

• ChemVA: Advancing Large Language Models on Chemical Reaction Diagrams Understanding [56.94419746470799]
  Large Language Models (LLMs) have revolutionized scientific text processing, but exhibit a significant capability gap when interpreting chemical reaction diagrams.<n>We propose the Chemical Visual Activation (ChemVA) framework, which employs a Visual Anchor mechanism to ground functional groups.<n>We evaluate our approach on OCRD-Bench, a newly constructed dataset featuring dense visual-semantic contexts.
  arXiv  Detail & Related papers  (2026-05-17T01:12:50Z)
• Deep Reasoning in General Purpose Agents via Structured Meta-Cognition [58.185853639335896]
  We introduce an inference-time approach for constructing task-specific scaffolds through structured meta-reasoning.<n>We instantiate this approach in a general-purpose agent that distributes complex tasks across more controlled reasoning threads.
  arXiv  Detail & Related papers  (2026-05-12T01:21:37Z)
• SLASH the Sink: Sharpening Structural Attention Inside LLMs [55.75072688988806]
  Large Language Models (LLMs) show remarkable semantic understanding but struggle with structural understanding when processing graph topologies in a serialized format.<n>We propose StructuraL Attention SHarpening (SLASH), which amplifies this internal structural understanding via a plug-and-play attention redistribution.
  arXiv  Detail & Related papers  (2026-05-11T12:59:07Z)
• Unmasking Reasoning Processes: A Process-aware Benchmark for Evaluating Structural Mathematical Reasoning in LLMs [20.82580343824728]
  Recent large language models (LLMs) achieve near-saturation accuracy on many established mathematical reasoning benchmarks.<n>This saturation stems from the dominance of template-based computation and shallow arithmetic decomposition.<n>We introduce ReasoningMath-Plus, a benchmark of 150 carefully curated problems explicitly designed to evaluate structural reasoning.
  arXiv  Detail & Related papers  (2026-01-31T07:09:17Z)
• Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study [40.143148197878354]
  We introduce FineLogic, a fine-grained evaluation framework that assesses logical reasoning across three dimensions.<n>We study how different supervision formats in fine-tuning shape reasoning abilities.<n>We find a key trade-off: natural language supervision excels at generalization, whereas symbolic supervision is superior at instilling structurally sound, atomic reasoning steps.
  arXiv  Detail & Related papers  (2025-06-05T09:34:12Z)
• Can LLMs Reason Structurally? An Evaluation via the Lens of Data Structures [21.390740746718947]
  We introduce DSR-Bench, the first benchmark to systematically evaluate large language models' structural reasoning.<n>The benchmark spans 20 data structures, 35 operations, and 4,140 synthetically generated problem instances with minimal contamination.
  arXiv  Detail & Related papers  (2025-05-29T23:24:53Z)
• Hallucination Detection in LLMs with Topological Divergence on Attention Graphs [60.83579255387347]
  Hallucination, i.e., generating factually incorrect content, remains a critical challenge for large language models.<n>We introduce TOHA, a TOpology-based HAllucination detector in the RAG setting.
  arXiv  Detail & Related papers  (2025-04-14T10:06:27Z)
• HiBench: Benchmarking LLMs Capability on Hierarchical Structure Reasoning [25.088407009353162]
  Existing benchmarks for structure reasoning mainly focus on horizontal and coordinate structures.<n>HiBench is the first framework spanning from initial structure generation to final proficiency assessment.<n>It consists of 30 tasks with varying hierarchical complexity, totaling 39,519 queries.
  arXiv  Detail & Related papers  (2025-03-02T14:25:37Z)
• GRS-QA -- Graph Reasoning-Structured Question Answering Dataset [50.223851616680754]
  We introduce the Graph Reasoning-Structured Question Answering dataset (GRS-QA), which includes both semantic contexts and reasoning structures for QA pairs. Unlike existing M-QA datasets, GRS-QA explicitly captures intricate reasoning pathways by constructing reasoning graphs. Our empirical analysis reveals that LLMs perform differently when handling questions with varying reasoning structures.
  arXiv  Detail & Related papers  (2024-11-01T05:14:03Z)
• Modeling Hierarchical Reasoning Chains by Linking Discourse Units and Key Phrases for Reading Comprehension [80.99865844249106]
  We propose a holistic graph network (HGN) which deals with context at both discourse level and word level, as the basis for logical reasoning. Specifically, node-level and type-level relations, which can be interpreted as bridges in the reasoning process, are modeled by a hierarchical interaction mechanism.
  arXiv  Detail & Related papers  (2023-06-21T07:34:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.