Is Self-Supervised Pretraining Good for Extrapolation in Molecular Property Prediction?

• URL: http://arxiv.org/abs/2308.08129v1
• Date: Wed, 16 Aug 2023 03:38:43 GMT
• Title: Is Self-Supervised Pretraining Good for Extrapolation in Molecular Property Prediction?
• Authors: Shun Takashige, Masatoshi Hanai, Toyotaro Suzumura, Limin Wang and Kenjiro Taura
• Abstract summary: In material science, the prediction of unobserved values, commonly referred to as extrapolation, is critical for property prediction. We propose an experimental framework for the demonstration and empirically reveal that while models were unable to accurately extrapolate absolute property values, self-supervised pretraining enables them to learn relative tendencies of unobserved property values.
• Score: 16.211138511816642
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The prediction of material properties plays a crucial role in the development and discovery of materials in diverse applications, such as batteries, semiconductors, catalysts, and pharmaceuticals. Recently, there has been a growing interest in employing data-driven approaches by using machine learning technologies, in combination with conventional theoretical calculations. In material science, the prediction of unobserved values, commonly referred to as extrapolation, is particularly critical for property prediction as it enables researchers to gain insight into materials beyond the limits of available data. However, even with the recent advancements in powerful machine learning models, accurate extrapolation is still widely recognized as a significantly challenging problem. On the other hand, self-supervised pretraining is a machine learning technique where a model is first trained on unlabeled data using relatively simple pretext tasks before being trained on labeled data for target tasks. As self-supervised pretraining can effectively utilize material data without observed property values, it has the potential to improve the model's extrapolation ability. In this paper, we clarify how such self-supervised pretraining can enhance extrapolation performance.We propose an experimental framework for the demonstration and empirically reveal that while models were unable to accurately extrapolate absolute property values, self-supervised pretraining enables them to learn relative tendencies of unobserved property values and improve extrapolation performance.

Related papers

• Imputation for prediction: beware of diminishing returns [12.424671213282256]
  Missing values are prevalent across various fields, posing challenges for training and deploying predictive models. Recent theoretical and empirical studies indicate that simple constant imputation can be consistent and competitive. This study aims at clarifying if and when investing in advanced imputation methods yields significantly better predictions.
  arXiv  Detail & Related papers  (2024-07-29T09:01:06Z)
• On Data Imbalance in Molecular Property Prediction with Pre-training [16.211138511816642]
  A technique called pre-training is used to improve the accuracy of machine learning models. Pre-training involves training the model on pretext task, which is different from the target task, before training the model on the target task. In this study, we propose an effective pre-training method that addresses the imbalance in input data.
  arXiv  Detail & Related papers  (2023-08-17T12:04:14Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• On the contribution of pre-trained models to accuracy and utility in modeling distributed energy resources [0.0]
  We evaluate the improvement in predictive accuracy due to pre-trained models, both with and without fine-tuning. We consider the question of fairness: do pre-trained models create equal improvements for heterogeneous agents, and how does this translate to downstream utility?
  arXiv  Detail & Related papers  (2023-02-22T22:29:40Z)
• Interpretable Self-Aware Neural Networks for Robust Trajectory Prediction [50.79827516897913]
  We introduce an interpretable paradigm for trajectory prediction that distributes the uncertainty among semantic concepts. We validate our approach on real-world autonomous driving data, demonstrating superior performance over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-11-16T06:28:20Z)
• Synthetic Model Combination: An Instance-wise Approach to Unsupervised Ensemble Learning [92.89846887298852]
  Consider making a prediction over new test data without any opportunity to learn from a training set of labelled data. Give access to a set of expert models and their predictions alongside some limited information about the dataset used to train them.
  arXiv  Detail & Related papers  (2022-10-11T10:20:31Z)
• Self-Distillation for Further Pre-training of Transformers [83.84227016847096]
  We propose self-distillation as a regularization for a further pre-training stage. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks.
  arXiv  Detail & Related papers  (2022-09-30T02:25:12Z)
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets [74.11825654535895]
  Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance. We study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
  arXiv  Detail & Related papers  (2021-09-08T10:39:57Z)
• Assigning Confidence to Molecular Property Prediction [1.015785232738621]
  Machine learning has emerged as a powerful strategy to learn from existing datasets and perform predictions on unseen molecules. We discuss popular strategies for predicting molecular properties relevant to drug design, their corresponding uncertainty sources and methods to quantify uncertainty and confidence.
  arXiv  Detail & Related papers  (2021-02-23T01:03:48Z)
• Statistical learning for accurate and interpretable battery lifetime prediction [1.738360170201861]
  We develop simple, accurate, and interpretable data-driven models for battery lifetime prediction. Our approaches can be used both to quickly train models for a new dataset and to benchmark the performance of more advanced machine learning methods.
  arXiv  Detail & Related papers  (2021-01-06T06:05:24Z)
• Value-driven Hindsight Modelling [68.658900923595]
  Value estimation is a critical component of the reinforcement learning (RL) paradigm. Model learning can make use of the rich transition structure present in sequences of observations, but this approach is usually not sensitive to the reward function. We develop an approach for representation learning in RL that sits in between these two extremes. This provides tractable prediction targets that are directly relevant for a task, and can thus accelerate learning the value function.
  arXiv  Detail & Related papers  (2020-02-19T18:10:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.