GP-MoLFormer: A Foundation Model For Molecular Generation

• URL: http://arxiv.org/abs/2405.04912v2
• Date: Mon, 31 Mar 2025 18:16:41 GMT
• Title: GP-MoLFormer: A Foundation Model For Molecular Generation
• Authors: Jerret Ross, Brian Belgodere, Samuel C. Hoffman, Vijil Chenthamarakshan, Jiri Navratil, Youssef Mroueh, Payel Das,
• Abstract summary: We extend the paradigm of training chemical language transformers on large-scale chemical datasets to generative tasks.<n>Specifically, we propose GP-MoLFormer, an autoregressive molecular string generator that is trained on more than 1.1B (billion) chemical SMILES.
• Score: 30.06169570297667
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Transformer-based models trained on large and general purpose datasets consisting of molecular strings have recently emerged as a powerful tool for successfully modeling various structure-property relations. Inspired by this success, we extend the paradigm of training chemical language transformers on large-scale chemical datasets to generative tasks in this work. Specifically, we propose GP-MoLFormer, an autoregressive molecular string generator that is trained on more than 1.1B (billion) chemical SMILES. GP-MoLFormer uses a 46.8M parameter transformer decoder model with linear attention and rotary positional encodings as the base architecture. GP-MoLFormer's utility is evaluated and compared with that of existing baselines on three different tasks: de novo generation, scaffold-constrained molecular decoration, and unconstrained property-guided optimization. While the first two are handled with no additional training, we propose a parameter-efficient fine-tuning method for the last task, which uses property-ordered molecular pairs as input. We call this new approach pair-tuning. Our results show GP-MoLFormer performs better or comparable with baselines across all three tasks, demonstrating its general utility for a variety of molecular generation tasks. We further report strong memorization of training data in GP-MoLFormer generations, which has so far remained unexplored for chemical language models. Our analyses reveal that training data memorization and novelty in generations are impacted by the quality and scale of the training data; duplication bias in training data can enhance memorization at the cost of lowering novelty. We further establish a scaling law relating inference compute and novelty in generations.

Related papers

• Metadata Conditioning Accelerates Language Model Pre-training [76.54265482251454]
  We propose a new method, termed Metadata Conditioning then Cooldown (MeCo) to incorporate additional learning cues during pre-training. MeCo significantly accelerates pre-training across different model scales (600M to 8B parameters) and training sources (C4, RefinedWeb, and DCLM) MeCo is remarkably simple, adds no computational overhead, and demonstrates promise in producing more capable and steerable language models.
  arXiv  Detail & Related papers  (2025-01-03T18:59:23Z)
• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• Generative Modeling of Molecular Dynamics Trajectories [12.255021091552441]
  We introduce generative modeling of molecular trajectories as a paradigm for learning flexible multi-task surrogate models of MD from data. We show such generative models can be adapted to diverse tasks such as forward simulation, transition path sampling, and trajectory upsampling.
  arXiv  Detail & Related papers  (2024-09-26T13:02:28Z)
• Assessing Non-Nested Configurations of Multifidelity Machine Learning for Quantum-Chemical Properties [0.0]
  Multifidelity machine learning (MFML) for quantum chemical (QC) properties has seen strong development in the recent years. This work assesses the use of non-nested training data for two of these multifidelity methods, namely MFML and optimized MFML.
  arXiv  Detail & Related papers  (2024-07-24T08:34:08Z)
• Uni-Mol2: Exploring Molecular Pretraining Model at Scale [27.172011090947823]
  We present Uni-Mol2, an innovative molecular pretraining model that integrates features at the atomic level, graph level, and geometry structure level. We successfully scale Uni-Mol2 to 1.1 billion parameters through pretraining on 800 million conformations, making it the largest molecular pretraining model to date.
  arXiv  Detail & Related papers  (2024-06-21T08:28:54Z)
• LLamol: A Dynamic Multi-Conditional Generative Transformer for De Novo Molecular Design [0.0]
  "LLamol" is a single novel generative transformer model based on the LLama 2 architecture. We demonstrate that the model adeptly handles single- and multi-conditional organic molecule generation with up to four conditions. In detail, we showcase the model's capability to utilize token sequences for conditioning, either individually or in combination with numerical properties.
  arXiv  Detail & Related papers  (2023-11-24T10:59:12Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective [53.300288393173204]
  Large Language Models (LLMs) have shown remarkable performance in various cross-modal tasks. In this work, we propose an In-context Few-Shot Molecule Learning paradigm for molecule-caption translation. We evaluate the effectiveness of MolReGPT on molecule-caption translation, including molecule understanding and text-based molecule generation.
  arXiv  Detail & Related papers  (2023-06-11T08:16:25Z)
• MolCPT: Molecule Continuous Prompt Tuning to Generalize Molecular Representation Learning [77.31492888819935]
  We propose a novel paradigm of "pre-train, prompt, fine-tune" for molecular representation learning, named molecule continuous prompt tuning (MolCPT) MolCPT defines a motif prompting function that uses the pre-trained model to project the standalone input into an expressive prompt. Experiments on several benchmark datasets show that MolCPT efficiently generalizes pre-trained GNNs for molecular property prediction.
  arXiv  Detail & Related papers  (2022-12-20T19:32:30Z)
• MolE: a molecular foundation model for drug discovery [0.2802437011072858]
  MolE is a molecular foundation model that adapts the DeBERTa architecture to be used on molecular graphs. We show that fine-tuning pretrained MolE achieves state-of-the-art results on 9 of the 22 ADMET tasks included in the Therapeutic Data Commons.
  arXiv  Detail & Related papers  (2022-11-03T21:22:05Z)
• ChemBERTa-2: Towards Chemical Foundation Models [0.0]
  We build a chemical foundation model, ChemBERTa-2, using the language of SMILES. In this work, we build upon ChemBERTa by optimizing the pretraining process. To our knowledge, the 77M set constitutes one of the largest datasets used for molecular pretraining to date.
  arXiv  Detail & Related papers  (2022-09-05T00:31:12Z)
• Retrieval-based Controllable Molecule Generation [63.44583084888342]
  We propose a new retrieval-based framework for controllable molecule generation. We use a small set of molecules to steer the pre-trained generative model towards synthesizing molecules that satisfy the given design criteria. Our approach is agnostic to the choice of generative models and requires no task-specific fine-tuning.
  arXiv  Detail & Related papers  (2022-08-23T17:01:16Z)
• Improving Small Molecule Generation using Mutual Information Machine [0.0]
  MolMIM is a probabilistic auto-encoder for small molecule drug discovery. We demonstrate MolMIM's superior generation as measured in terms of validity, uniqueness, and novelty. We then utilize CMA-ES, a naive black-box and gradient free search algorithm, over MolMIM's latent space for the task of property guided molecule optimization.
  arXiv  Detail & Related papers  (2022-08-18T18:32:48Z)
• Molecular Attributes Transfer from Non-Parallel Data [57.010952598634944]
  We formulate molecular optimization as a style transfer problem and present a novel generative model that could automatically learn internal differences between two groups of non-parallel data. Experiments on two molecular optimization tasks, toxicity modification and synthesizability improvement, demonstrate that our model significantly outperforms several state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-11-30T06:10:22Z)
• Reprogramming Language Models for Molecular Representation Learning [65.00999660425731]
  We propose Representation Reprogramming via Dictionary Learning (R2DL) for adversarially reprogramming pretrained language models for molecular learning tasks. The adversarial program learns a linear transformation between a dense source model input space (language data) and a sparse target model input space (e.g., chemical and biological molecule data) using a k-SVD solver. R2DL achieves the baseline established by state of the art toxicity prediction models trained on domain-specific data and outperforms the baseline in a limited training-data setting.
  arXiv  Detail & Related papers  (2020-12-07T05:50:27Z)
• MIMOSA: Multi-constraint Molecule Sampling for Molecule Optimization [51.00815310242277]
  generative models and reinforcement learning approaches made initial success, but still face difficulties in simultaneously optimizing multiple drug properties. We propose the MultI-constraint MOlecule SAmpling (MIMOSA) approach, a sampling framework to use input molecule as an initial guess and sample molecules from the target distribution.
  arXiv  Detail & Related papers  (2020-10-05T20:18:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.