MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures

• URL: http://arxiv.org/abs/2603.28550v1
• Date: Mon, 30 Mar 2026 15:11:17 GMT
• Title: MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures
• Abstract summary: MarkushGrapher-2 is an end-to-end approach for the multimodal recognition of chemical structures in documents.<n>We introduce an automatic pipeline for constructing a large-scale dataset of real-world Markush structures.<n>Our approach substantially outperforms state-of-the-art models in multimodal Markush structure recognition.
• Score: 2.7994101180516133
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatically extracting chemical structures from documents is essential for the large-scale analysis of the literature in chemistry. Automatic pipelines have been developed to recognize molecules represented either in figures or in text independently. However, methods for recognizing chemical structures from multimodal descriptions (Markush structures) lag behind in precision and cannot be used for automatic large-scale processing. In this work, we present MarkushGrapher-2, an end-to-end approach for the multimodal recognition of chemical structures in documents. First, our method employs a dedicated OCR model to extract text from chemical images. Second, the text, image, and layout information are jointly encoded through a Vision-Text-Layout encoder and an Optical Chemical Structure Recognition vision encoder. Finally, the resulting encodings are effectively fused through a two-stage training strategy and used to auto-regressively generate a representation of the Markush structure. To address the lack of training data, we introduce an automatic pipeline for constructing a large-scale dataset of real-world Markush structures. In addition, we present IP5-M, a large manually-annotated benchmark of real-world Markush structures, designed to advance research on this challenging task. Extensive experiments show that our approach substantially outperforms state-of-the-art models in multimodal Markush structure recognition, while maintaining strong performance in molecule structure recognition. Code, models, and datasets are released publicly.

Related papers

• MarkushGlyph and OCSRGlyph: Improved Chemical Structure Recognition [3.930866087901632]
  Chemical structures appear in patents and the scientific literature as images.<n>For programmatic usage, such as indexing in databases or constructing machine learning model training sets, they must be transformed into line notations.<n>In this work, we treat both tasks as an image-to-text translation problem.<n>We propose OCSRGlyph, a state-of-the-art OCSR model, improving performance over prior methods by carefully considering stereochemistry.<n>For the Markush task, we introduce MarkushGlyph, a vision-language model that reads the entire Markush structure as an image.
  arXiv  Detail & Related papers  (2026-07-30T17:03:05Z)
• MonkeyOCR v1.5 Technical Report: Unlocking Robust Document Parsing for Complex Patterns [80.05126590825121]
  MonkeyOCR v1.5 is a unified vision-language framework that enhances both layout understanding and content recognition.<n>To address complex table structures, we propose a visual consistency-based reinforcement learning scheme.<n>Two specialized modules, Image-Decoupled Table Parsing and Type-Guided Table Merging, are introduced to enable reliable parsing of tables.
  arXiv  Detail & Related papers  (2025-11-13T15:12:17Z)
• RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning [51.393018266721576]
  We propose the RxnCaption framework for the task of chemical Reaction Diagram Parsing (RxnDP)<n>Our framework reformulates the traditional coordinate prediction driven parsing process into an image captioning problem.<n>We introduce a strategy termed "BBox and Index as Visual Prompt" (BIVP), which uses our state-of-the-art molecular detector, MolYOLO, to pre-draw molecular bounding boxes and indices directly onto the input image.
  arXiv  Detail & Related papers  (2025-11-04T09:08:44Z)
• GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular Structure Recognition [60.76623665324548]
  GTR-Mol-VLM is a novel framework featuring two key innovations.<n>It emulates human reasoning by incrementally parsing molecular graphs through sequential atom-bond predictions.<n>MolRec-Bench is the first benchmark designed for a fine-grained evaluation of graph-parsing accuracy in OCSR.
  arXiv  Detail & Related papers  (2025-06-09T08:47:10Z)
• SubGrapher: Visual Fingerprinting of Chemical Structures [45.2191459937801]
  SubGrapher is a method for the visual fingerprinting of chemical structure images.<n>It identifies functional groups and carbon backbones, constructing a substructure-based fingerprint.<n>Our approach is evaluated against state-of-the-art OCSR and fingerprinting methods.
  arXiv  Detail & Related papers  (2025-04-28T11:45:46Z)
• MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures [47.41884299076947]
  MarkushGrapher is a multi-modal approach for recognizing Markush structures in documents.<n>We propose a synthetic data generation pipeline that produces a wide range of realistic Markush structures.<n>M2S is the first annotated benchmark of real-world Markush structures.
  arXiv  Detail & Related papers  (2025-03-20T12:40:38Z)
• OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models [58.45517851437422]
  Visually-situated text parsing (VsTP) has recently seen notable advancements, driven by the growing demand for automated document understanding.<n>Existing solutions often rely on task-specific architectures and objectives for individual tasks.<n>In this paper, we introduce Omni V2, a universal model that unifies VsTP typical tasks, including text spotting, key information extraction, table recognition, and layout analysis.
  arXiv  Detail & Related papers  (2025-02-22T09:32:01Z)
• MolParser: End-to-end Visual Recognition of Molecule Structures in the Wild [17.846545370594452]
  We present Mol, a novel end-to-end optical chemical structure recognition method.<n>We use a SMILES encoding rule to annotate Mol-7M, the largest annotated molecular image dataset.<n>We trained an end-to-end molecular image captioning model, Mol, using a curriculum learning approach.
  arXiv  Detail & Related papers  (2024-11-17T15:00:09Z)
• Exploring Optimal Transport-Based Multi-Grained Alignments for Text-Molecule Retrieval [24.061535843472427]
  We introduce the Optimal TRansport-based Multi-grained Alignments model (ORMA) ORMA is a novel approach that facilitates multi-grained alignments between textual descriptions and molecules. Experimental results on the ChEBI-20 and PCdes datasets demonstrate that ORMA significantly outperforms existing state-of-the-art (SOTA) models.
  arXiv  Detail & Related papers  (2024-11-04T06:30:52Z)
• MolGrapher: Graph-based Visual Recognition of Chemical Structures [50.13749978547401]
  We introduce MolGrapher to recognize chemical structures visually. We treat all candidate atoms and bonds as nodes and put them in a graph. We classify atom and bond nodes in the graph with a Graph Neural Network.
  arXiv  Detail & Related papers  (2023-08-23T16:16:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.