Pre-training via Denoising for Molecular Property Prediction

• URL: http://arxiv.org/abs/2206.00133v1
• Date: Tue, 31 May 2022 22:28:34 GMT
• Title: Pre-training via Denoising for Molecular Property Prediction
• Authors: Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, Jonathan Godwin
• Abstract summary: We describe a pre-training technique that utilizes large datasets of 3D molecular structures at equilibrium. Inspired by recent advances in noise regularization, our pre-training objective is based on denoising.
• Score: 53.409242538744444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many important problems involving molecular property prediction from 3D structures have limited data, posing a generalization challenge for neural networks. In this paper, we describe a pre-training technique that utilizes large datasets of 3D molecular structures at equilibrium to learn meaningful representations for downstream tasks. Inspired by recent advances in noise regularization, our pre-training objective is based on denoising. Relying on the well-known link between denoising autoencoders and score-matching, we also show that the objective corresponds to learning a molecular force field -- arising from approximating the physical state distribution with a mixture of Gaussians -- directly from equilibrium structures. Our experiments demonstrate that using this pre-training objective significantly improves performance on multiple benchmarks, achieving a new state-of-the-art on the majority of targets in the widely used QM9 dataset. Our analysis then provides practical insights into the effects of different factors -- dataset sizes, model size and architecture, and the choice of upstream and downstream datasets -- on pre-training.

Related papers

• On Machine Learning Approaches for Protein-Ligand Binding Affinity Prediction [2.874893537471256]
  This study evaluates the performance of classical tree-based models and advanced neural networks in protein-ligand binding affinity prediction. We show that combining 2D and 3D model strengths improves active learning outcomes beyond current state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-07-15T13:06:00Z)
• Learning with Noisy Foundation Models [95.50968225050012]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy [55.014926694758195]
  Entropy and mutual information in neural networks provide rich information on the learning process. We leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. We show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data.
  arXiv  Detail & Related papers  (2023-12-04T01:32:42Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• 3D Denoisers are Good 2D Teachers: Molecular Pretraining via Denoising and Cross-Modal Distillation [65.35632020653291]
  We propose D&D, a self-supervised molecular representation learning framework that pretrains a 2D graph encoder by distilling representations from a 3D denoiser. We show that D&D can infer 3D information based on the 2D graph and shows superior performance and label-efficiency against other baselines.
  arXiv  Detail & Related papers  (2023-09-08T01:36:58Z)
• Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
  We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data. Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
  arXiv  Detail & Related papers  (2023-07-12T16:28:21Z)
• Deep learning for full-field ultrasonic characterization [7.120879473925905]
  This study takes advantage of recent advances in machine learning to establish a physics-based data analytic platform. Two logics, namely the direct inversion and physics-informed neural networks (PINNs), are explored.
  arXiv  Detail & Related papers  (2023-01-06T05:01:05Z)
• 3D Equivariant Molecular Graph Pretraining [42.957880677779556]
  We tackle 3D molecular pretraining in a complete and novel sense. We first propose to adopt an equivariant energy-based model as the backbone for pretraining, which enjoys the merit of fulfilling the symmetry of 3D space. We evaluate our model pretrained from a large-scale 3D dataset GEOM-QM9 on two challenging 3D benchmarks: MD17 and QM9.
  arXiv  Detail & Related papers  (2022-07-18T16:26:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.