Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modeling

• URL: http://arxiv.org/abs/2410.08024v1
• Date: Thu, 10 Oct 2024 15:20:30 GMT
• Title: Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modeling
• Authors: Alessio Fallani, Ramil Nugmanov, Jose Arjona-Medina, Jörg Kurt Wegner, Alexandre Tkatchenko, Kostiantyn Chernichenko,
• Abstract summary: We evaluate the impact of pretraining Graph Transformer architectures on atom-level quantum-mechanical features. We find that models pretrained on atomic quantum mechanical properties capture more low-frequency laplacian eigenmodes.
• Score: 38.53065398127086
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We evaluate the impact of pretraining Graph Transformer architectures on atom-level quantum-mechanical features for the modeling of absorption, distribution, metabolism, excretion, and toxicity (ADMET) properties of drug-like compounds. We compare this pretraining strategy with two others: one based on molecular quantum properties (specifically the HOMO-LUMO gap) and one using a self-supervised atom masking technique. After fine-tuning on Therapeutic Data Commons ADMET datasets, we evaluate the performance improvement in the different models observing that models pretrained with atomic quantum mechanical properties produce in general better results. We then analyse the latent representations and observe that the supervised strategies preserve the pretraining information after finetuning and that different pretrainings produce different trends in latent expressivity across layers. Furthermore, we find that models pretrained on atomic quantum mechanical properties capture more low-frequency laplacian eigenmodes of the input graph via the attention weights and produce better representations of atomic environments within the molecule. Application of the analysis to a much larger non-public dataset for microsomal clearance illustrates generalizability of the studied indicators. In this case the performances of the models are in accordance with the representation analysis and highlight, especially for the case of masking pretraining and atom-level quantum property pretraining, how model types with similar performance on public benchmarks can have different performances on large scale pharmaceutical data.

Related papers

• Pre-trained Molecular Language Models with Random Functional Group Masking [54.900360309677794]
  We propose a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular atoms. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities.
  arXiv  Detail & Related papers  (2024-11-03T01:56:15Z)
• Analysis of Atom-level pretraining with Quantum Mechanics (QM) data for Graph Neural Networks Molecular property models [0.0]
  We show how atom-level pretraining with quantum mechanics (QM) data can mitigate violations of assumptions regarding the distributional similarity between training and test data. This is the first time that hidden state molecular representations are analyzed to compare the effects of molecule-level and atom-level pretraining on QM data.
  arXiv  Detail & Related papers  (2024-05-23T17:51:05Z)
• Machine Learning Small Molecule Properties in Drug Discovery [44.62264781248437]
  We review a wide range of properties, including binding affinities, solubility, and ADMET (Absorption, Distribution, Metabolism, Excretion, and Toxicity) We discuss existing popular descriptors and embeddings, such as chemical fingerprints and graph-based neural networks. Finally, techniques to provide an understanding of model predictions, especially for critical decision-making in drug discovery are assessed.
  arXiv  Detail & Related papers  (2023-08-02T22:18:41Z)
• Atomic and Subgraph-aware Bilateral Aggregation for Molecular Representation Learning [57.670845619155195]
  We introduce a new model for molecular representation learning called the Atomic and Subgraph-aware Bilateral Aggregation (ASBA) ASBA addresses the limitations of previous atom-wise and subgraph-wise models by incorporating both types of information. Our method offers a more comprehensive way to learn representations for molecular property prediction and has broad potential in drug and material discovery applications.
  arXiv  Detail & Related papers  (2023-05-22T00:56:00Z)
• PEMP: Leveraging Physics Properties to Enhance Molecular Property Prediction [33.715410811008375]
  We propose Physics properties Enhanced Molecular Property prediction (PEMP) to utilize relations between molecular properties revealed by previous physics theory and physical chemistry studies. We design two different methods for PEMP, respectively based on multi-task learning and transfer learning. Experimental results on public benchmark MoleculeNet show that the proposed methods have the ability to outperform corresponding state-of-the-art models.
  arXiv  Detail & Related papers  (2022-10-18T07:40:58Z)
• KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction [13.55018269009361]
  We introduce Knowledge-guided Pre-training of Graph Transformer (KPGT), a novel self-supervised learning framework for molecular graph representation learning. KPGT can offer superior performance over current state-of-the-art methods on several molecular property prediction tasks.
  arXiv  Detail & Related papers  (2022-06-02T08:22:14Z)
• Pre-training via Denoising for Molecular Property Prediction [53.409242538744444]
  We describe a pre-training technique that utilizes large datasets of 3D molecular structures at equilibrium. Inspired by recent advances in noise regularization, our pre-training objective is based on denoising.
  arXiv  Detail & Related papers  (2022-05-31T22:28:34Z)
• Molecular Attributes Transfer from Non-Parallel Data [57.010952598634944]
  We formulate molecular optimization as a style transfer problem and present a novel generative model that could automatically learn internal differences between two groups of non-parallel data. Experiments on two molecular optimization tasks, toxicity modification and synthesizability improvement, demonstrate that our model significantly outperforms several state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-11-30T06:10:22Z)
• Do Large Scale Molecular Language Representations Capture Important Structural Information? [31.76876206167457]
  We present molecular embeddings obtained by training an efficient transformer encoder model, referred to as MoLFormer. Experiments show that the learned molecular representation performs competitively, when compared to graph-based and fingerprint-based supervised learning baselines.
  arXiv  Detail & Related papers  (2021-06-17T14:33:55Z)
• Few-Shot Graph Learning for Molecular Property Prediction [46.60746023179724]
  We propose Meta-MGNN, a novel model for few-shot molecular property prediction. To exploit unlabeled molecular information, Meta-MGNN further incorporates molecular structure, attribute based self-supervised modules and self-attentive task weights. Extensive experiments on two public multi-property datasets demonstrate that Meta-MGNN outperforms a variety of state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-02-16T01:55:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.