MolXPT: Wrapping Molecules with Text for Generative Pre-training

• URL: http://arxiv.org/abs/2305.10688v2
• Date: Fri, 26 May 2023 04:35:46 GMT
• Title: MolXPT: Wrapping Molecules with Text for Generative Pre-training
• Authors: Zequn Liu, Wei Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang and Tie-Yan Liu
• Abstract summary: MolXPT is a unified language model of text and molecules pre-trained on SMILES wrapped by text. MolXPT outperforms strong baselines of molecular property prediction on MoleculeNet.
• Score: 141.0924452870112
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative pre-trained Transformer (GPT) has demonstrates its great success in natural language processing and related techniques have been adapted into molecular modeling. Considering that text is the most important record for scientific discovery, in this paper, we propose MolXPT, a unified language model of text and molecules pre-trained on SMILES (a sequence representation of molecules) wrapped by text. Briefly, we detect the molecule names in each sequence and replace them to the corresponding SMILES. In this way, the SMILES could leverage the information from surrounding text, and vice versa. The above wrapped sequences, text sequences from PubMed and SMILES sequences from PubChem are all fed into a language model for pre-training. Experimental results demonstrate that MolXPT outperforms strong baselines of molecular property prediction on MoleculeNet, performs comparably to the best model in text-molecule translation while using less than half of its parameters, and enables zero-shot molecular generation without finetuning.

Related papers

• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• Chemical Language Model Linker: blending text and molecules with modular adapters [2.2667044928324747]
  We propose a lightweight adapter-based strategy named Chemical Language Model Linker (ChemLML) ChemLML blends the two single domain models and obtains conditional molecular generation from text descriptions. We find that the choice of molecular representation used within ChemLML, SMILES versus SELFIES, has a strong influence on conditional molecular generation performance.
  arXiv  Detail & Related papers  (2024-10-26T13:40:13Z)
• Text-Guided Multi-Property Molecular Optimization with a Diffusion Language Model [77.50732023411811]
  We propose a text-guided multi-property molecular optimization method utilizing transformer-based diffusion language model (TransDLM) TransDLM leverages standardized chemical nomenclature as semantic representations of molecules and implicitly embeds property requirements into textual descriptions. Our approach surpasses state-of-the-art methods in optimizing molecular structural similarity and enhancing chemical properties on the benchmark dataset.
  arXiv  Detail & Related papers  (2024-10-17T14:30:27Z)
• UniMoT: Unified Molecule-Text Language Model with Discrete Token Representation [35.51027934845928]
  We introduce UniMoT, a Unified Molecule-Text LLM adopting a tokenizer-based architecture. A Vector Quantization-driven tokenizer transforms molecules into sequences of molecule tokens with causal dependency. UniMoT emerges as a multi-modal generalist capable of performing both molecule-to-text and text-to-molecule tasks.
  arXiv  Detail & Related papers  (2024-08-01T18:31:31Z)
• MolTRES: Improving Chemical Language Representation Learning for Molecular Property Prediction [14.353313239109337]
  MolTRES is a novel chemical language representation learning framework. It incorporates generator-discriminator training, allowing the model to learn from more challenging examples. Our model outperforms existing state-of-the-art models on popular molecular property prediction tasks.
  arXiv  Detail & Related papers  (2024-07-09T01:14:28Z)
• Instruction Multi-Constraint Molecular Generation Using a Teacher-Student Large Language Model [49.64512917330373]
  We introduce a multi-constraint molecular generation large language model, TSMMG, akin to a student. To train TSMMG, we construct a large set of text-molecule pairs by extracting molecular knowledge from these 'teachers' We experimentally show that TSMMG remarkably performs in generating molecules meeting complex, natural language-described property requirements.
  arXiv  Detail & Related papers  (2024-03-20T02:15:55Z)
• Text-Guided Molecule Generation with Diffusion Language Model [23.170313481324598]
  We propose the Text-Guided Molecule Generation with Diffusion Language Model (TGM-DLM) TGM-DLM updates token embeddings within the SMILES string collectively and iteratively, using a two-phase diffusion generation process. We demonstrate that TGM-DLM outperforms MolT5-Base, an autoregressive model, without the need for additional data resources.
  arXiv  Detail & Related papers  (2024-02-20T14:29:02Z)
• Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective [53.300288393173204]
  Large Language Models (LLMs) have shown remarkable performance in various cross-modal tasks. In this work, we propose an In-context Few-Shot Molecule Learning paradigm for molecule-caption translation. We evaluate the effectiveness of MolReGPT on molecule-caption translation, including molecule understanding and text-based molecule generation.
  arXiv  Detail & Related papers  (2023-06-11T08:16:25Z)
• A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language [63.60376252491507]
  We propose a molecular multimodal foundation model which is pretrained from molecular graphs and their semantically related textual data. We believe that our model would have a broad impact on AI-empowered fields across disciplines such as biology, chemistry, materials, environment, and medicine.
  arXiv  Detail & Related papers  (2022-09-12T00:56:57Z)
• Translation between Molecules and Natural Language [43.518805086280466]
  We present a self-supervised learning framework for pretraining models on a vast amount of unlabeled natural language text and molecule strings. $textbfMolT5$ allows for new, useful, and challenging analogs of traditional vision-language tasks, such as molecule captioning and text-based de novo molecule generation.
  arXiv  Detail & Related papers  (2022-04-25T17:48:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.