Transformers for molecular property prediction: Lessons learned from the past five years

• URL: http://arxiv.org/abs/2404.03969v1
• Date: Fri, 5 Apr 2024 09:05:37 GMT
• Title: Transformers for molecular property prediction: Lessons learned from the past five years
• Authors: Afnan Sultan, Jochen Sieg, Miriam Mathea, Andrea Volkamer,
• Abstract summary: We analyze the currently available models and explore key questions that arise when training and fine-tuning a transformer model for MPP. We address the challenges in comparing different models, emphasizing the need for standardized data splitting and robust statistical analysis.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Molecular Property Prediction (MPP) is vital for drug discovery, crop protection, and environmental science. Over the last decades, diverse computational techniques have been developed, from using simple physical and chemical properties and molecular fingerprints in statistical models and classical machine learning to advanced deep learning approaches. In this review, we aim to distill insights from current research on employing transformer models for MPP. We analyze the currently available models and explore key questions that arise when training and fine-tuning a transformer model for MPP. These questions encompass the choice and scale of the pre-training data, optimal architecture selections, and promising pre-training objectives. Our analysis highlights areas not yet covered in current research, inviting further exploration to enhance the field's understanding. Additionally, we address the challenges in comparing different models, emphasizing the need for standardized data splitting and robust statistical analysis.

Related papers

• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• Advancements in Molecular Property Prediction: A Survey of Single and Multimodal Approaches [1.0446041735532203]
  Molecular Property Prediction (MPP) plays a pivotal role across diverse domains, spanning drug discovery, material science, and environmental chemistry. Recent years have witnessed remarkable strides in MPP, fueled by the exponential growth of chemical data and the evolution of artificial intelligence. This article explores recent AI/based approaches in MPP, focusing on both single and multiple modality representation techniques.
  arXiv  Detail & Related papers  (2024-08-18T12:49:52Z)
• Molecule Design by Latent Prompt Transformer [76.2112075557233]
  This work explores the challenging problem of molecule design by framing it as a conditional generative modeling task. We propose a novel generative model comprising three components: (1) a latent vector with a learnable prior distribution; (2) a molecule generation model based on a causal Transformer, which uses the latent vector as a prompt; and (3) a property prediction model that predicts a molecule's target properties and/or constraint values using the latent prompt.
  arXiv  Detail & Related papers  (2024-02-27T03:33:23Z)
• Transferring a molecular foundation model for polymer property predictions [3.067983186439152]
  Self-supervised pretraining of transformer models requires large-scale datasets. We show that using transformers pretrained on small molecules and fine-tuned on polymer properties achieve comparable accuracy to those trained on augmented polymer datasets.
  arXiv  Detail & Related papers  (2023-10-25T19:55:00Z)
• Dynamic Molecular Graph-based Implementation for Biophysical Properties Prediction [9.112532782451233]
  We propose a novel approach based on the transformer model utilizing GNNs for characterizing dynamic features of protein-ligand interactions. Our message passing transformer pre-trains on a set of molecular dynamic data based off of physics-based simulations to learn coordinate construction and make binding probability and affinity predictions.
  arXiv  Detail & Related papers  (2022-12-20T04:21:19Z)
• Calibration and generalizability of probabilistic models on low-data chemical datasets with DIONYSUS [0.0]
  We perform an extensive study of the calibration and generalizability of probabilistic machine learning models on small chemical datasets. We analyse the quality of their predictions and uncertainties in a variety of tasks (binary, regression) and datasets. We offer practical insights into model and feature choice for modelling small chemical datasets, a common scenario in new chemical experiments.
  arXiv  Detail & Related papers  (2022-12-03T08:19:06Z)
• Retrieval-based Controllable Molecule Generation [63.44583084888342]
  We propose a new retrieval-based framework for controllable molecule generation. We use a small set of molecules to steer the pre-trained generative model towards synthesizing molecules that satisfy the given design criteria. Our approach is agnostic to the choice of generative models and requires no task-specific fine-tuning.
  arXiv  Detail & Related papers  (2022-08-23T17:01:16Z)
• Pre-training Transformers for Molecular Property Prediction Using Reaction Prediction [0.0]
  Transfer learning has had a tremendous impact in fields like Computer Vision and Natural Language Processing. We present a pre-training procedure for molecular representation learning using reaction data. We show a statistically significant positive effect on 5 of the 12 tasks compared to a non-pre-trained baseline model.
  arXiv  Detail & Related papers  (2022-07-06T14:51:38Z)
• Improving Molecular Representation Learning with Metric Learning-enhanced Optimal Transport [49.237577649802034]
  We develop a novel optimal transport-based algorithm termed MROT to enhance their generalization capability for molecular regression problems. MROT significantly outperforms state-of-the-art models, showing promising potential in accelerating the discovery of new substances.
  arXiv  Detail & Related papers  (2022-02-13T04:56:18Z)
• Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data [66.11139091362078]
  We provide the first model selection results on large pretrained Transformers from Huggingface using generalization metrics. Despite their niche status, we find that metrics derived from the heavy-tail (HT) perspective are particularly useful in NLP tasks.
  arXiv  Detail & Related papers  (2022-02-06T20:07:35Z)
• Prediction of liquid fuel properties using machine learning models with Gaussian processes and probabilistic conditional generative learning [56.67751936864119]
  The present work aims to construct cheap-to-compute machine learning (ML) models to act as closure equations for predicting the physical properties of alternative fuels. Those models can be trained using the database from MD simulations and/or experimental measurements in a data-fusion-fidelity approach. The results show that ML models can predict accurately the fuel properties of a wide range of pressure and temperature conditions.
  arXiv  Detail & Related papers  (2021-10-18T14:43:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.