MolSight: A Graph-Aware Vision-Language Model for Unified Chemical Image Understanding

• URL: http://arxiv.org/abs/2607.01982v1
• Date: Thu, 02 Jul 2026 10:13:19 GMT
• Title: MolSight: A Graph-Aware Vision-Language Model for Unified Chemical Image Understanding
• Abstract summary: MolSight is a graph-aware vision-language model framework designed to enhance the understanding of molecular images.<n>Our experiments demonstrate that MolSight significantly outperforms existing VLMs, molecular LLMs, and specialized tools across multiple chemical visual understanding tasks.
• Score: 25.768840369692608
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Using molecular large language models (LLMs) as a unified framework for understanding molecular structures and functions is emerging as a new trend in tasks such as molecular design and drug discovery. However, these models struggle to fully capture the visual representation of molecular structures, limiting their potential. While existing molecular vision-language models (VLMs) show promise, they still face challenges in structural alignment and lack the necessary topological modeling for accurate molecular understanding. To address this, we propose MolSight, a graph-aware vision-language model framework designed to enhance the understanding of molecular images by VLMs. MolSight integrates a Molecular Topology Module to inject chemical-bond adjacency information into vision tokens, and a Molecular Grounding Module to align visual features with chemical symbolic semantics. Our experiments demonstrate that MolSight significantly outperforms existing VLMs, molecular LLMs, and specialized tools across multiple chemical visual understanding tasks, achieving a new level of molecular image reasoning.

Related papers

• KnowMol: Advancing Molecular Large Language Models with Multi-Level Chemical Knowledge [73.51130155601824]
  We introduce KnowMol-100K, a large-scale dataset with 100K fine-grained molecular annotations across multiple levels.<n>We also propose chemically-informative molecular representation, effectively addressing limitations in existing molecular representation strategies.<n>KnowMol achieves superior performance across molecular understanding and generation tasks.
  arXiv  Detail & Related papers  (2025-10-22T11:23:58Z)
• $\text{M}^{2}$LLM: Multi-view Molecular Representation Learning with Large Language Models [59.125833618091846]
  We propose a multi-view framework that integrates three perspectives: the molecular structure view, the molecular task view, and the molecular rules view.<n>Experiments demonstrate that $textM2$LLM achieves state-of-the-art performance on multiple benchmarks across classification and regression tasks.
  arXiv  Detail & Related papers  (2025-08-12T05:46:47Z)
• Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model [52.84455878597969]
  Mol-LLaMA is a large molecular language model that grasps the general knowledge centered on molecules.<n>To improve molecular understanding, we propose a module that integrates complementary information from different molecular encoders.
  arXiv  Detail & Related papers  (2025-02-19T05:49:10Z)
• Knowledge-aware contrastive heterogeneous molecular graph learning [77.94721384862699]
  We propose a paradigm shift by encoding molecular graphs into Heterogeneous Molecular Graph Learning (KCHML)<n>KCHML conceptualizes molecules through three distinct graph views-molecular, elemental, and pharmacological-enhanced by heterogeneous molecular graphs and a dual message-passing mechanism.<n>This design offers a comprehensive representation for property prediction, as well as for downstream tasks such as drug-drug interaction (DDI) prediction.
  arXiv  Detail & Related papers  (2025-02-17T11:53:58Z)
• MolMetaLM: a Physicochemical Knowledge-Guided Molecular Meta Language Model [19.458584012046646]
  We propose a novel physicochemical knowledge-guided molecular meta language framework MolMetaLM. We design a molecule-specialized meta language paradigm, formatted as multiple S,P,O> knowledge triples sharing the same S (i.e., molecule) By introducing different molecular knowledge and noises, the meta language paradigm generates tens of thousands of pretraining tasks.
  arXiv  Detail & Related papers  (2024-11-23T09:27:38Z)
• FARM: Functional Group-Aware Representations for Small Molecules [55.281754551202326]
  We introduce Functional Group-Aware Representations for Small Molecules (FARM)<n>FARM is a novel model designed to bridge the gap between SMILES, natural language, and molecular graphs.<n>We evaluate FARM on the MoleculeNet dataset, where it achieves state-of-the-art performance on 11 out of 13 tasks.
  arXiv  Detail & Related papers  (2024-10-02T23:04:58Z)
• Learning Multi-view Molecular Representations with Structured and Unstructured Knowledge [14.08112359246334]
  We present MV-Mol, a representation learning model that harvests multi-view molecular expertise from chemical structures, unstructured knowledge from biomedical texts, and structured knowledge from knowledge graphs. We show that MV-Mol provides improved representations that substantially benefit molecular property prediction.
  arXiv  Detail & Related papers  (2024-06-14T08:48:10Z)
• MultiModal-Learning for Predicting Molecular Properties: A Framework Based on Image and Graph Structures [2.5563339057415218]
  MolIG is a novel MultiModaL molecular pre-training framework for predicting molecular properties based on Image and Graph structures. It amalgamates the strengths of both molecular representation forms. It exhibits enhanced performance in downstream tasks pertaining to molecular property prediction within benchmark groups.
  arXiv  Detail & Related papers  (2023-11-28T10:28:35Z)
• A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language [63.60376252491507]
  We propose a molecular multimodal foundation model which is pretrained from molecular graphs and their semantically related textual data. We believe that our model would have a broad impact on AI-empowered fields across disciplines such as biology, chemistry, materials, environment, and medicine.
  arXiv  Detail & Related papers  (2022-09-12T00:56:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.