Clarifying Trust of Materials Property Predictions using Neural Networks with Distribution-Specific Uncertainty Quantification

• URL: http://arxiv.org/abs/2302.02595v1
• Date: Mon, 6 Feb 2023 07:03:02 GMT
• Title: Clarifying Trust of Materials Property Predictions using Neural Networks with Distribution-Specific Uncertainty Quantification
• Authors: Cameron Gruich, Varun Madhavan, Yixin Wang, Bryan Goldsmith
• Abstract summary: Uncertainty (UQ) methods allow estimation of the trustworthiness of machine learning (ML) model predictions. Here, we investigate different UQ methods applied to predict energies of molecules on alloys from the Open Catalyst 2020 dataset. Evidential regression is demonstrated to be a powerful approach for rapidly obtaining, competitively trustworthy UQ estimates.
• Score: 16.36620228609086
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It is critical that machine learning (ML) model predictions be trustworthy for high-throughput catalyst discovery approaches. Uncertainty quantification (UQ) methods allow estimation of the trustworthiness of an ML model, but these methods have not been well explored in the field of heterogeneous catalysis. Herein, we investigate different UQ methods applied to a crystal graph convolutional neural network (CGCNN) to predict adsorption energies of molecules on alloys from the Open Catalyst 2020 (OC20) dataset, the largest existing heterogeneous catalyst dataset. We apply three UQ methods to the adsorption energy predictions, namely k-fold ensembling, Monte Carlo dropout, and evidential regression. The effectiveness of each UQ method is assessed based on accuracy, sharpness, dispersion, calibration, and tightness. Evidential regression is demonstrated to be a powerful approach for rapidly obtaining tunable, competitively trustworthy UQ estimates for heterogeneous catalysis applications when using neural networks. Recalibration of model uncertainties is shown to be essential in practical screening applications of catalysts using uncertainties.

Related papers

• Achieving Well-Informed Decision-Making in Drug Discovery: A Comprehensive Calibration Study using Neural Network-Based Structure-Activity Models [4.619907534483781]
  computational models that predict drug-target interactions are valuable tools to accelerate the development of new therapeutic agents. However, such models can be poorly calibrated, which results in unreliable uncertainty estimates. We show that combining post hoc calibration method with well-performing uncertainty quantification approaches can boost model accuracy and calibration.
  arXiv  Detail & Related papers  (2024-07-19T10:29:00Z)
• Improved Uncertainty Estimation of Graph Neural Network Potentials Using Engineered Latent Space Distances [1.9286144773392733]
  We show that uncertainty quantification for relaxed energy calculations is more complex than uncertainty quantification for other kinds of molecular property prediction. We propose that distribution-free techniques are more useful tools for assessing calibration, recalibrating, and developing uncertainty prediction methods for GNNs performing relaxed energy calculations.
  arXiv  Detail & Related papers  (2024-07-15T15:59:39Z)
• Balancing Molecular Information and Empirical Data in the Prediction of Physico-Chemical Properties [8.649679686652648]
  We propose a general method for combining molecular descriptors with representation learning. The proposed hybrid model exploits chemical structure information using graph neural networks. It automatically detects cases where structure-based predictions are unreliable, in which case it corrects them by representation-learning based predictions.
  arXiv  Detail & Related papers  (2024-06-12T10:51:00Z)
• Decoupling of neural network calibration measures [45.70855737027571]
  We investigate the coupling of different neural network calibration measures with a special focus on the Area Under Sparsification Error curve (AUSE) metric. We conclude that the current methodologies leave a degree of freedom, which prevents a unique model for the homologation of safety-critical functionalities.
  arXiv  Detail & Related papers  (2024-06-04T15:21:37Z)
• Ensemble models outperform single model uncertainties and predictions for operator-learning of hypersonic flows [43.148818844265236]
  Training scientific machine learning (SciML) models on limited high-fidelity data offers one approach to rapidly predict behaviors for situations that have not been seen before. High-fidelity data is itself in limited quantity to validate all outputs of the SciML model in unexplored input space. We extend a DeepONet using three different uncertainty mechanisms: mean-variance estimation, evidential uncertainty, and ensembling.
  arXiv  Detail & Related papers  (2023-10-31T18:07:29Z)
• Machine Learning Small Molecule Properties in Drug Discovery [44.62264781248437]
  We review a wide range of properties, including binding affinities, solubility, and ADMET (Absorption, Distribution, Metabolism, Excretion, and Toxicity) We discuss existing popular descriptors and embeddings, such as chemical fingerprints and graph-based neural networks. Finally, techniques to provide an understanding of model predictions, especially for critical decision-making in drug discovery are assessed.
  arXiv  Detail & Related papers  (2023-08-02T22:18:41Z)
• Uncertainty Quantification for Molecular Property Predictions with Graph Neural Architecture Search [2.711812013460678]
  We introduce AutoGNNUQ, an automated uncertainty quantification (UQ) approach for molecular property prediction. Our approach employs variance decomposition to separate data (aleatoric) and model (epistemic) uncertainties, providing valuable insights for reducing them. AutoGNNUQ has broad applicability in domains such as drug discovery and materials science, where accurate uncertainty quantification is crucial for decision-making.
  arXiv  Detail & Related papers  (2023-07-19T20:03:42Z)
• Uncertainty quantification for predictions of atomistic neural networks [0.0]
  This paper explores the value of uncertainty quantification on predictions for trained neural networks (NNs) on quantum chemical reference data. The architecture of the PhysNet NN was suitably modified and the resulting model was evaluated with different metrics to quantify calibration, quality of predictions, and whether prediction error and the predicted uncertainty can be correlated.
  arXiv  Detail & Related papers  (2022-07-14T13:39:43Z)
• Conditional-Flow NeRF: Accurate 3D Modelling with Reliable Uncertainty Quantification [44.598503284186336]
  Conditional-Flow NeRF (CF-NeRF) is a novel probabilistic framework to incorporate uncertainty quantification into NeRF-based approaches. CF-NeRF learns a distribution over all possible radiance fields modelling which is used to quantify the uncertainty associated with the modelled scene.
  arXiv  Detail & Related papers  (2022-03-18T23:26:20Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.