Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables

• URL: http://arxiv.org/abs/2506.11375v1
• Date: Fri, 13 Jun 2025 00:45:41 GMT
• Title: Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables
• Authors: Yitong Zhou, Mingyue Cheng, Qingyang Mao, Yucong Luo, Qi Liu, Yupeng Li, Xiaohan Zhang, Deguang Liu, Xin Li, Enhong Chen,
• Abstract summary: ChemTable is a large-scale benchmark of real-world chemical tables curated from the experimental sections of literature.<n>ChemTable includes expert-annotated cell polygons, logical layouts, and domain-specific labels, including reagents, catalysts, yields, and graphical components.<n>We evaluated a range of representative multimodal models, including both open-source and closed-source models, on ChemTable and reported a series of findings with practical and conceptual insights.
• Score: 48.39080455781475
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Chemical tables encode complex experimental knowledge through symbolic expressions, structured variables, and embedded molecular graphics. Existing benchmarks largely overlook this multimodal and domain-specific complexity, limiting the ability of multimodal large language models to support scientific understanding in chemistry. In this work, we introduce ChemTable, a large-scale benchmark of real-world chemical tables curated from the experimental sections of literature. ChemTable includes expert-annotated cell polygons, logical layouts, and domain-specific labels, including reagents, catalysts, yields, and graphical components and supports two core tasks: (1) Table Recognition, covering structure parsing and content extraction; and (2) Table Understanding, encompassing both descriptive and reasoning-oriented question answering grounded in table structure and domain semantics. We evaluated a range of representative multimodal models, including both open-source and closed-source models, on ChemTable and reported a series of findings with practical and conceptual insights. Although models show reasonable performance on basic layout parsing, they exhibit substantial limitations on both descriptive and inferential QA tasks compared to human performance, and we observe significant performance gaps between open-source and closed-source models across multiple dimensions. These results underscore the challenges of chemistry-aware table understanding and position ChemTable as a rigorous and realistic benchmark for advancing scientific reasoning.

Related papers

• QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistry [12.18966912295507]
  QCBench is a benchmark comprising 350 computational chemistry problems across 7 chemistry subfields.<n>Each problem focuses on pure calculations rooted in real-world chemical vertical fields.<n> Evaluations on 19 LLMs demonstrate a consistent performance degradation with increasing task complexity.
  arXiv  Detail & Related papers  (2025-08-03T08:55:42Z)
• A Multi-Agent System Enables Versatile Information Extraction from the Chemical Literature [8.306442315850878]
  We develop a multimodal large language model (MLLM)-based multi-agent system for robust and automated chemical information extraction.<n>Our system achieved an F1 score of 80.8% on a benchmark dataset of sophisticated multimodal chemical reaction graphics from the literature.
  arXiv  Detail & Related papers  (2025-07-27T11:16:57Z)
• ChemActor: Enhancing Automated Extraction of Chemical Synthesis Actions with LLM-Generated Data [53.78763789036172]
  We present ChemActor, a fully fine-tuned large language model (LLM) as a chemical executor to convert between unstructured experimental procedures and structured action sequences.<n>This framework integrates a data selection module that selects data based on distribution divergence, with a general-purpose LLM, to generate machine-executable actions from a single molecule input.<n>Experiments on reaction-to-description (R2D) and description-to-action (D2A) tasks demonstrate that ChemActor achieves state-of-the-art performance, outperforming the baseline model by 10%.
  arXiv  Detail & Related papers  (2025-06-30T05:11:19Z)
• TableMoE: Neuro-Symbolic Routing for Structured Expert Reasoning in Multimodal Table Understanding [3.404552731440374]
  TableMoE is a neuro-symbolic Mixture-of-Connector-Experts (MoCE) architecture specifically designed for robust, structured reasoning over multimodal table data.<n>TableMoE features an innovative Neuro-Symbolic Routing mechanism, which predicts latent semantic token roles and dynamically routes table elements to specialized experts.<n>For evaluation, we curate and release four challenging WildStruct benchmarks, designed specifically to stress-test models under real-world multimodal degradation and structural complexity.
  arXiv  Detail & Related papers  (2025-06-26T15:41:34Z)
• ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area [50.15254966969718]
  We introduce textbfChemVLM, an open-source chemical multimodal large language model for chemical applications.<n>ChemVLM is trained on a carefully curated bilingual dataset that enhances its ability to understand both textual and visual chemical information.<n>We benchmark ChemVLM against a range of open-source and proprietary multimodal large language models on various tasks.
  arXiv  Detail & Related papers  (2024-08-14T01:16:40Z)
• A Large Encoder-Decoder Family of Foundation Models For Chemical Language [1.1073864511426255]
  This paper introduces a large encoder-decoder chemical foundation models pre-trained on a curated dataset of 91 million SMILES samples sourced from PubChem. Our experiments across multiple benchmark datasets validate the capacity of the proposed model in providing state-of-the-art results for different tasks.
  arXiv  Detail & Related papers  (2024-07-24T20:30:39Z)
• ScholarChemQA: Unveiling the Power of Language Models in Chemical Research Question Answering [54.80411755871931]
  Question Answering (QA) effectively evaluates language models' reasoning and knowledge depth. Chemical QA plays a crucial role in both education and research by effectively translating complex chemical information into readily understandable format. This dataset reflects typical real-world challenges, including an imbalanced data distribution and a substantial amount of unlabeled data that can be potentially useful. We introduce a QAMatch model, specifically designed to effectively answer chemical questions by fully leveraging our collected data.
  arXiv  Detail & Related papers  (2024-07-24T01:46:55Z)
• TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy [81.76462101465354]
  We present a novel large vision-hugging model, TabPedia, equipped with a concept synergy mechanism. This unified framework allows TabPedia to seamlessly integrate VTU tasks, such as table detection, table structure recognition, table querying, and table question answering. To better evaluate the VTU task in real-world scenarios, we establish a new and comprehensive table VQA benchmark, ComTQA.
  arXiv  Detail & Related papers  (2024-06-03T13:54:05Z)
• ChemMiner: A Large Language Model Agent System for Chemical Literature Data Mining [56.15126714863963]
  ChemMiner is an end-to-end framework for extracting chemical data from literature.<n>ChemMiner incorporates three specialized agents: a text analysis agent for coreference mapping, a multimodal agent for non-textual information extraction, and a synthesis analysis agent for data generation.<n> Experimental results demonstrate reaction identification rates comparable to human chemists while significantly reducing processing time, with high accuracy, recall, and F1 scores.
  arXiv  Detail & Related papers  (2024-02-20T13:21:46Z)
• DiSCoMaT: Distantly Supervised Composition Extraction from Tables in Materials Science Articles [25.907266860321727]
  We define a novel NLP task of extracting compositions of materials from tables in materials science papers. We release a training dataset comprising 4,408 distantly supervised tables, along with 1,475 manually annotated dev and test tables. We show that DISCOMAT outperforms recent table processing architectures by significant margins.
  arXiv  Detail & Related papers  (2022-07-03T17:11:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.