BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models

• URL: http://arxiv.org/abs/2505.01912v1
• Date: Sat, 03 May 2025 19:51:23 GMT
• Title: BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models
• Authors: Evan R. Antoniuk, Shehtab Zaman, Tal Ben-Nun, Peggy Li, James Diffenderfer, Busra Demirci, Obadiah Smolenski, Tim Hsu, Anna M. Hiszpanski, Kenneth Chiu, Bhavya Kailkhura, Brian Van Essen,
• Abstract summary: We present BOOM, $boldsymbolb$enchmarks for $boldsymbolo$f-distribution $boldsymbolm$olecular property predictions.<n>We evaluate more than 140 combinations of models and property prediction tasks to benchmark deep learning models on their OOD performance.<n>Overall, we do not find any existing models that achieve strong OOD generalization across all tasks.
• Score: 12.92528375287641
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Advances in deep learning and generative modeling have driven interest in data-driven molecule discovery pipelines, whereby machine learning (ML) models are used to filter and design novel molecules without requiring prohibitively expensive first-principles simulations. Although the discovery of novel molecules that extend the boundaries of known chemistry requires accurate out-of-distribution (OOD) predictions, ML models often struggle to generalize OOD. Furthermore, there are currently no systematic benchmarks for molecular OOD prediction tasks. We present BOOM, $\boldsymbol{b}$enchmarks for $\boldsymbol{o}$ut-$\boldsymbol{o}$f-distribution $\boldsymbol{m}$olecular property predictions -- a benchmark study of property-based out-of-distribution models for common molecular property prediction models. We evaluate more than 140 combinations of models and property prediction tasks to benchmark deep learning models on their OOD performance. Overall, we do not find any existing models that achieve strong OOD generalization across all tasks: even the top performing model exhibited an average OOD error 3x larger than in-distribution. We find that deep learning models with high inductive bias can perform well on OOD tasks with simple, specific properties. Although chemical foundation models with transfer and in-context learning offer a promising solution for limited training data scenarios, we find that current foundation models do not show strong OOD extrapolation capabilities. We perform extensive ablation experiments to highlight how OOD performance is impacted by data generation, pre-training, hyperparameter optimization, model architecture, and molecular representation. We propose that developing ML models with strong OOD generalization is a new frontier challenge in chemical ML model development. This open-source benchmark will be made available on Github.

Related papers

• Machine Learning-Based Prediction of Metal-Organic Framework Materials: A Comparative Analysis of Multiple Models [2.089191490381739]
  Metal-organic frameworks (MOFs) have emerged as promising materials for various applications.<n>This study presents a comprehensive investigation of machine learning approaches for predicting MOF material properties.
  arXiv  Detail & Related papers  (2025-07-06T18:10:00Z)
• Jet: A Modern Transformer-Based Normalizing Flow [62.2573739835562]
  We revisit the design of the coupling-based normalizing flow models by carefully ablating prior design choices.<n>We achieve state-of-the-art quantitative and qualitative performance with a much simpler architecture.
  arXiv  Detail & Related papers  (2024-12-19T18:09:42Z)
• Can OOD Object Detectors Learn from Foundation Models? [56.03404530594071]
  Out-of-distribution (OOD) object detection is a challenging task due to the absence of open-set OOD data. Inspired by recent advancements in text-to-image generative models, we study the potential of generative models trained on large-scale open-set data to synthesize OOD samples. We introduce SyncOOD, a simple data curation method that capitalizes on the capabilities of large foundation models.
  arXiv  Detail & Related papers  (2024-09-08T17:28:22Z)
• Out-of-distribution materials property prediction using adversarial learning based fine-tuning [0.0]
  We propose an adversarial learning based targeting finetuning approach to make the model adapted to a particular dataset. Our experiments demonstrate the success of our CAL algorithm with its high effectiveness in ML with limited samples.
  arXiv  Detail & Related papers  (2024-08-17T21:22:21Z)
• Structure-based out-of-distribution (OOD) materials property prediction: a benchmark study [1.3711992220025948]
  We present a benchmark study of structure-based graph neural networks (GNNs) for extrapolative OOD materials property prediction. Our experiments show that current state-of-the-art GNN algorithms significantly underperform for the OOD property prediction tasks. We identify the sources of CGCNN, ALIGNN, and DeeperGATGNN's significantly more robust OOD performance than those of the current best models.
  arXiv  Detail & Related papers  (2024-01-16T01:03:39Z)
• End-to-End Reinforcement Learning of Koopman Models for Economic Nonlinear Model Predictive Control [45.84205238554709]
  We present a method for reinforcement learning of Koopman surrogate models for optimal performance as part of (e)NMPC. We show that the end-to-end trained models outperform those trained using system identification in (e)NMPC.
  arXiv  Detail & Related papers  (2023-08-03T10:21:53Z)
• Pseudo-OOD training for robust language models [78.15712542481859]
  OOD detection is a key component of a reliable machine-learning model for any industry-scale application. We propose POORE - POsthoc pseudo-Ood REgularization, that generates pseudo-OOD samples using in-distribution (IND) data. We extensively evaluate our framework on three real-world dialogue systems, achieving new state-of-the-art in OOD detection.
  arXiv  Detail & Related papers  (2022-10-17T14:32:02Z)
• How robust are pre-trained models to distribution shift? [82.08946007821184]
  We show how spurious correlations affect the performance of popular self-supervised learning (SSL) and auto-encoder based models (AE) We develop a novel evaluation scheme with the linear head trained on out-of-distribution (OOD) data, to isolate the performance of the pre-trained models from a potential bias of the linear head used for evaluation.
  arXiv  Detail & Related papers  (2022-06-17T16:18:28Z)
• On the Generalization and Adaption Performance of Causal Models [99.64022680811281]
  Differentiable causal discovery has proposed to factorize the data generating process into a set of modules. We study the generalization and adaption performance of such modular neural causal models. Our analysis shows that the modular neural causal models outperform other models on both zero and few-shot adaptation in low data regimes.
  arXiv  Detail & Related papers  (2022-06-09T17:12:32Z)
• BEDS-Bench: Behavior of EHR-models under Distributional Shift--A Benchmark [21.040754460129854]
  We release BEDS-Bench, a benchmark for quantifying the behavior of ML models over EHR data under OOD settings. We evaluate several learning algorithms under BEDS-Bench and find that all of them show poor generalization performance under distributional shift in general.
  arXiv  Detail & Related papers  (2021-07-17T05:53:24Z)
• Generative Temporal Difference Learning for Infinite-Horizon Prediction [101.59882753763888]
  We introduce the $gamma$-model, a predictive model of environment dynamics with an infinite probabilistic horizon. We discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors.
  arXiv  Detail & Related papers  (2020-10-27T17:54:12Z)
• Goal-directed Generation of Discrete Structures with Conditional Generative Models [85.51463588099556]
  We introduce a novel approach to directly optimize a reinforcement learning objective, maximizing an expected reward. We test our methodology on two tasks: generating molecules with user-defined properties and identifying short python expressions which evaluate to a given target value.
  arXiv  Detail & Related papers  (2020-10-05T20:03:13Z)
• Model Embedding Model-Based Reinforcement Learning [4.566180616886624]
  Model-based reinforcement learning (MBRL) has shown its advantages in sample-efficiency over model-free reinforcement learning (MFRL) Despite the impressive results it achieves, it still faces a trade-off between the ease of data generation and model bias. We propose a simple and elegant model-embedding model-based reinforcement learning (MEMB) algorithm in the framework of the probabilistic reinforcement learning.
  arXiv  Detail & Related papers  (2020-06-16T15:10:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.