RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning

• URL: http://arxiv.org/abs/2511.02384v1
• Date: Tue, 04 Nov 2025 09:08:44 GMT
• Title: RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning
• Abstract summary: We propose the RxnCaption framework for the task of chemical Reaction Diagram Parsing (RxnDP)<n>Our framework reformulates the traditional coordinate prediction driven parsing process into an image captioning problem.<n>We introduce a strategy termed "BBox and Index as Visual Prompt" (BIVP), which uses our state-of-the-art molecular detector, MolYOLO, to pre-draw molecular bounding boxes and indices directly onto the input image.
• Score: 51.393018266721576
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale chemical reaction datasets are crucial for AI research in chemistry. However, existing chemical reaction data often exist as images within papers, making them not machine-readable and unusable for training machine learning models. In response to this challenge, we propose the RxnCaption framework for the task of chemical Reaction Diagram Parsing (RxnDP). Our framework reformulates the traditional coordinate prediction driven parsing process into an image captioning problem, which Large Vision-Language Models (LVLMs) handle naturally. We introduce a strategy termed "BBox and Index as Visual Prompt" (BIVP), which uses our state-of-the-art molecular detector, MolYOLO, to pre-draw molecular bounding boxes and indices directly onto the input image. This turns the downstream parsing into a natural-language description problem. Extensive experiments show that the BIVP strategy significantly improves structural extraction quality while simplifying model design. We further construct the RxnCaption-11k dataset, an order of magnitude larger than prior real-world literature benchmarks, with a balanced test subset across four layout archetypes. Experiments demonstrate that RxnCaption-VL achieves state-of-the-art performance on multiple metrics. We believe our method, dataset, and models will advance structured information extraction from chemical literature and catalyze broader AI applications in chemistry. We will release data, models, and code on GitHub.

Related papers

• ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding [18.366771283768344]
  We introduce ChemVLR, a chemical VLM designed to prioritize reasoning within the perception process.<n>Unlike conventional chemical VLMs, ChemVLR analyzes visual inputs in a fine-grained manner.<n>ChemVLR produces explicit and interpretable reasoning paths for complex visual chemical problems.
  arXiv  Detail & Related papers  (2026-04-08T05:01:59Z)
• Molecular Identifier Visual Prompt and Verifiable Reinforcement Learning for Chemical Reaction Diagram Parsing [52.825281124618535]
  Reaction diagram parsing (RxnDP) is critical for extracting chemical synthesis information from literature.<n>Recent Vision-Language Models (VLMs) have emerged as a promising paradigm to automate this complex visual reasoning task.<n>This work enhances VLM-based RxnDP from two complementary perspectives: prompting representation and learning paradigms.
  arXiv  Detail & Related papers  (2026-03-16T09:17:05Z)
• ChemActor: Enhancing Automated Extraction of Chemical Synthesis Actions with LLM-Generated Data [53.78763789036172]
  We present ChemActor, a fully fine-tuned large language model (LLM) as a chemical executor to convert between unstructured experimental procedures and structured action sequences.<n>This framework integrates a data selection module that selects data based on distribution divergence, with a general-purpose LLM, to generate machine-executable actions from a single molecule input.<n>Experiments on reaction-to-description (R2D) and description-to-action (D2A) tasks demonstrate that ChemActor achieves state-of-the-art performance, outperforming the baseline model by 10%.
  arXiv  Detail & Related papers  (2025-06-30T05:11:19Z)
• Interpretable Deep Learning for Polar Mechanistic Reaction Prediction [43.95903801494905]
  We introduce PMechRP (Polar Mechanistic Reaction Predictor), a system that trains machine learning models on the PMechDB dataset.<n>We train compare a range of machine learning models, including transformer-based, graph-based and two-step siamese architectures.<n>Our best-performing approach was a hybrid model, which combines a 5-ensemble of Chemformer models with a two-step Siamese framework.
  arXiv  Detail & Related papers  (2025-04-22T02:31:23Z)
• Towards Large-scale Chemical Reaction Image Parsing via a Multimodal Large Language Model [4.860497022313892]
  We introduce the Reaction Image Multimodal large language model (RxnIM) to parse chemical reaction images into machine-readable data.<n> RxnIM extracts key chemical components from reaction images and interprets the textual content that describes reaction conditions.<n>Our approach achieves excellent performance, with an average F1 score of 88% on various benchmarks, surpassing literature methods by 5%.
  arXiv  Detail & Related papers  (2025-03-11T08:11:23Z)
• ChemMiner: A Large Language Model Agent System for Chemical Literature Data Mining [56.15126714863963]
  ChemMiner is an end-to-end framework for extracting chemical data from literature.<n>ChemMiner incorporates three specialized agents: a text analysis agent for coreference mapping, a multimodal agent for non-textual information extraction, and a synthesis analysis agent for data generation.<n> Experimental results demonstrate reaction identification rates comparable to human chemists while significantly reducing processing time, with high accuracy, recall, and F1 scores.
  arXiv  Detail & Related papers  (2024-02-20T13:21:46Z)
• Predictive Chemistry Augmented with Text Retrieval [37.59545092901872]
  We introduce TextReact, a novel method that directly augments predictive chemistry with texts retrieved from the literature. TextReact retrieves text descriptions relevant for a given chemical reaction, and then aligns them with the molecular representation of the reaction. We empirically validate the framework on two chemistry tasks: reaction condition recommendation and one-step retrosynthesis.
  arXiv  Detail & Related papers  (2023-12-08T07:40:59Z)
• MolGrapher: Graph-based Visual Recognition of Chemical Structures [50.13749978547401]
  We introduce MolGrapher to recognize chemical structures visually. We treat all candidate atoms and bonds as nodes and put them in a graph. We classify atom and bond nodes in the graph with a Graph Neural Network.
  arXiv  Detail & Related papers  (2023-08-23T16:16:11Z)
• GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning [71.89623260998934]
  This study investigates the feasibility of employing natural language instructions to accomplish molecule-related tasks in a zero-shot setting. Existing molecule-text models perform poorly in this setting due to inadequate treatment of instructions and limited capacity for graphs. We propose GIMLET, which unifies language models for both graph and text data.
  arXiv  Detail & Related papers  (2023-05-28T18:27:59Z)
• Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction [2.5655440962401617]
  We describe a novel Graph2SMILES model that combines the power of Transformer models for text generation with the permutation invariance of molecular graph encoders. As an end-to-end architecture, Graph2SMILES can be used as a drop-in replacement for the Transformer in any task involving molecule(s)-to-molecule(s) transformations.
  arXiv  Detail & Related papers  (2021-10-19T01:23:15Z)
• Retrosynthesis Prediction with Conditional Graph Logic Network [118.70437805407728]
  Computer-aided retrosynthesis is finding renewed interest from both chemistry and computer science communities. We propose a new approach to this task using the Conditional Graph Logic Network, a conditional graphical model built upon graph neural networks.
  arXiv  Detail & Related papers  (2020-01-06T05:36:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.