Two-Stage Pretraining for Molecular Property Prediction in the Wild

• URL: http://arxiv.org/abs/2411.03537v1
• Date: Tue, 05 Nov 2024 22:36:17 GMT
• Title: Two-Stage Pretraining for Molecular Property Prediction in the Wild
• Authors: Kevin Tirta Wijaya, Minghao Guo, Michael Sun, Hans-Peter Seidel, Wojciech Matusik, Vahid Babaei,
• Abstract summary: We introduce MoleVers, a versatile pretrained model designed for various types of molecular property prediction in the wild. MoleVers learns representations from large unlabeled datasets via masked atom prediction and dynamic denoising. In the second stage, MoleVers is further pretrained using auxiliary labels obtained with inexpensive computational methods.
• Score: 38.31911435361748
• License:
• Abstract: Accurate property prediction is crucial for accelerating the discovery of new molecules. Although deep learning models have achieved remarkable success, their performance often relies on large amounts of labeled data that are expensive and time-consuming to obtain. Thus, there is a growing need for models that can perform well with limited experimentally-validated data. In this work, we introduce MoleVers, a versatile pretrained model designed for various types of molecular property prediction in the wild, i.e., where experimentally-validated molecular property labels are scarce. MoleVers adopts a two-stage pretraining strategy. In the first stage, the model learns molecular representations from large unlabeled datasets via masked atom prediction and dynamic denoising, a novel task enabled by a new branching encoder architecture. In the second stage, MoleVers is further pretrained using auxiliary labels obtained with inexpensive computational methods, enabling supervised learning without the need for costly experimental data. This two-stage framework allows MoleVers to learn representations that generalize effectively across various downstream datasets. We evaluate MoleVers on a new benchmark comprising 22 molecular datasets with diverse types of properties, the majority of which contain 50 or fewer training labels reflecting real-world conditions. MoleVers achieves state-of-the-art results on 20 out of the 22 datasets, and ranks second among the remaining two, highlighting its ability to bridge the gap between data-hungry models and real-world conditions where practically-useful labels are scarce.

Related papers

• MoleculeCLA: Rethinking Molecular Benchmark via Computational Ligand-Target Binding Analysis [18.940529282539842]
  We construct a large-scale and precise molecular representation dataset of approximately 140,000 small molecules. Our dataset offers significant physicochemical interpretability to guide model development and design. We believe this dataset will serve as a more accurate and reliable benchmark for molecular representation learning.
  arXiv  Detail & Related papers  (2024-06-13T02:50:23Z)
• Data-Efficient Molecular Generation with Hierarchical Textual Inversion [48.816943690420224]
  We introduce Hierarchical textual Inversion for Molecular generation (HI-Mol), a novel data-efficient molecular generation method. HI-Mol is inspired by the importance of hierarchical information, e.g., both coarse- and fine-grained features, in understanding the molecule distribution. Compared to the conventional textual inversion method in the image domain using a single-level token embedding, our multi-level token embeddings allow the model to effectively learn the underlying low-shot molecule distribution.
  arXiv  Detail & Related papers  (2024-05-05T08:35:23Z)
• Bi-level Contrastive Learning for Knowledge-Enhanced Molecule Representations [55.42602325017405]
  We propose a novel method called GODE, which takes into account the two-level structure of individual molecules. By pre-training two graph neural networks (GNNs) on different graph structures, combined with contrastive learning, GODE fuses molecular structures with their corresponding knowledge graph substructures. When fine-tuned across 11 chemical property tasks, our model outperforms existing benchmarks, registering an average ROC-AUC uplift of 13.8% for classification tasks and an average RMSE/MAE enhancement of 35.1% for regression tasks.
  arXiv  Detail & Related papers  (2023-06-02T15:49:45Z)
• Supervised Pretraining for Molecular Force Fields and Properties Prediction [16.86839767858162]
  We propose to pretrain neural networks on a dataset of 86 millions of molecules with atom charges and 3D geometries as inputs and molecular energies as labels. Experiments show that, compared to training from scratch, fine-tuning the pretrained model can significantly improve the performance for seven molecular property prediction tasks and two force field tasks.
  arXiv  Detail & Related papers  (2022-11-23T08:36:50Z)
• MolE: a molecular foundation model for drug discovery [0.2802437011072858]
  MolE is a molecular foundation model that adapts the DeBERTa architecture to be used on molecular graphs. We show that fine-tuning pretrained MolE achieves state-of-the-art results on 9 of the 22 ADMET tasks included in the Therapeutic Data Commons.
  arXiv  Detail & Related papers  (2022-11-03T21:22:05Z)
• Unraveling Key Elements Underlying Molecular Property Prediction: A Systematic Study [27.56700461408765]
  Key elements underlying molecular property prediction remain largely unexplored. We conduct an extensive evaluation of representative models using various representations on the MoleculeNet datasets. In total, we have trained 62,820 models, including 50,220 models on fixed representations, 4,200 models on SMILES sequences and 8,400 models on molecular graphs.
  arXiv  Detail & Related papers  (2022-09-26T14:07:59Z)
• Tyger: Task-Type-Generic Active Learning for Molecular Property Prediction [121.97742787439546]
  How to accurately predict the properties of molecules is an essential problem in AI-driven drug discovery. To reduce annotation cost, deep Active Learning methods are developed to select only the most representative and informative data for annotating. We propose a Task-type-generic active learning framework (termed Tyger) that is able to handle different types of learning tasks in a unified manner.
  arXiv  Detail & Related papers  (2022-05-23T12:56:12Z)
• Few-Shot Graph Learning for Molecular Property Prediction [46.60746023179724]
  We propose Meta-MGNN, a novel model for few-shot molecular property prediction. To exploit unlabeled molecular information, Meta-MGNN further incorporates molecular structure, attribute based self-supervised modules and self-attentive task weights. Extensive experiments on two public multi-property datasets demonstrate that Meta-MGNN outperforms a variety of state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-02-16T01:55:34Z)
• ASGN: An Active Semi-supervised Graph Neural Network for Molecular Property Prediction [61.33144688400446]
  We propose a novel framework called Active Semi-supervised Graph Neural Network (ASGN) by incorporating both labeled and unlabeled molecules. In the teacher model, we propose a novel semi-supervised learning method to learn general representation that jointly exploits information from molecular structure and molecular distribution. At last, we proposed a novel active learning strategy in terms of molecular diversities to select informative data during the whole framework learning.
  arXiv  Detail & Related papers  (2020-07-07T04:22:39Z)
• Self-Supervised Graph Transformer on Large-Scale Molecular Data [73.3448373618865]
  We propose a novel framework, GROVER, for molecular representation learning. GROVER can learn rich structural and semantic information of molecules from enormous unlabelled molecular data. We pre-train GROVER with 100 million parameters on 10 million unlabelled molecules -- the biggest GNN and the largest training dataset in molecular representation learning.
  arXiv  Detail & Related papers  (2020-06-18T08:37:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.