Challenging reaction prediction models to generalize to novel chemistry

• URL: http://arxiv.org/abs/2501.06669v1
• Date: Sat, 11 Jan 2025 23:49:14 GMT
• Title: Challenging reaction prediction models to generalize to novel chemistry
• Authors: John Bradshaw, Anji Zhang, Babak Mahjour, David E. Graff, Marwin H. S. Segler, Connor W. Coley,
• Abstract summary: We report a series of evaluations of a prototypical SMILES-based deep learning model. First, we illustrate how performance on randomly sampled datasets is overly optimistic compared to performance when generalizing to new patents or new authors. Second, we conduct time splits that evaluate how models perform when tested on reactions published in years after those in their training set, mimicking real-world deployment.
• Score: 12.33727805025678
• License:
• Abstract: Deep learning models for anticipating the products of organic reactions have found many use cases, including validating retrosynthetic pathways and constraining synthesis-based molecular design tools. Despite compelling performance on popular benchmark tasks, strange and erroneous predictions sometimes ensue when using these models in practice. The core issue is that common benchmarks test models in an in-distribution setting, whereas many real-world uses for these models are in out-of-distribution settings and require a greater degree of extrapolation. To better understand how current reaction predictors work in out-of-distribution domains, we report a series of more challenging evaluations of a prototypical SMILES-based deep learning model. First, we illustrate how performance on randomly sampled datasets is overly optimistic compared to performance when generalizing to new patents or new authors. Second, we conduct time splits that evaluate how models perform when tested on reactions published in years after those in their training set, mimicking real-world deployment. Finally, we consider extrapolation across reaction classes to reflect what would be required for the discovery of novel reaction types. This panel of tasks can reveal the capabilities and limitations of today's reaction predictors, acting as a crucial first step in the development of tomorrow's next-generation models capable of reaction discovery.

Related papers

• Chimera: Accurate retrosynthesis prediction by ensembling models with diverse inductive biases [3.885174353072695]
  Planning and conducting chemical syntheses remains a major bottleneck in the discovery of functional small molecules. Inspired by how chemists use different strategies to ideate reactions, we propose Chimera: a framework for building highly accurate reaction models.
  arXiv  Detail & Related papers  (2024-12-06T18:55:19Z)
• Beyond Major Product Prediction: Reproducing Reaction Mechanisms with Machine Learning Models Trained on a Large-Scale Mechanistic Dataset [10.968137261042715]
  Mechanistic understanding of organic reactions can facilitate reaction development, impurity prediction, and in principle, reaction discovery. While several machine learning models have sought to address the task of predicting reaction products, their extension to predicting reaction mechanisms has been impeded by the lack of a corresponding mechanistic dataset. We construct such a dataset by imputing intermediates between experimentally reported reactants and products using expert reaction templates and train several machine learning models on the resulting dataset of 5,184,184 elementary steps.
  arXiv  Detail & Related papers  (2024-03-07T15:26:23Z)
• Predictive Churn with the Set of Good Models [64.05949860750235]
  We study the effect of conflicting predictions over the set of near-optimal machine learning models. We present theoretical results on the expected churn between models within the Rashomon set. We show how our approach can be used to better anticipate, reduce, and avoid churn in consumer-facing applications.
  arXiv  Detail & Related papers  (2024-02-12T16:15:25Z)
• Holistic chemical evaluation reveals pitfalls in reaction prediction models [0.3065062372337749]
  We propose a new assessment scheme that builds on current approaches, steering towards a more holistic evaluation. ChoRISO is a curated dataset along with multiple tailored splits to recreate chemically relevant scenarios. Our work paves the way towards robust prediction models that can ultimately accelerate chemical discovery.
  arXiv  Detail & Related papers  (2023-12-14T14:54:28Z)
• Beyond the Typical: Modeling Rare Plausible Patterns in Chemical Reactions by Leveraging Sequential Mixture-of-Experts [42.9784548283531]
  Generative models like Transformer and VAE have typically been employed to predict the reaction product. We propose organizing the mapping space between reactants and electron redistribution patterns in a divide-and-conquer manner.
  arXiv  Detail & Related papers  (2023-10-07T03:18:26Z)
• MetaRF: Differentiable Random Forest for Reaction Yield Prediction with a Few Trails [58.47364143304643]
  In this paper, we focus on the reaction yield prediction problem. We first put forth MetaRF, an attention-based differentiable random forest model specially designed for the few-shot yield prediction. To improve the few-shot learning performance, we further introduce a dimension-reduction based sampling method.
  arXiv  Detail & Related papers  (2022-08-22T06:40:13Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• Back2Future: Leveraging Backfill Dynamics for Improving Real-time Predictions in Future [73.03458424369657]
  In real-time forecasting in public health, data collection is a non-trivial and demanding task. 'Backfill' phenomenon and its effect on model performance has been barely studied in the prior literature. We formulate a novel problem and neural framework Back2Future that aims to refine a given model's predictions in real-time.
  arXiv  Detail & Related papers  (2021-06-08T14:48:20Z)
• Goal-directed Generation of Discrete Structures with Conditional Generative Models [85.51463588099556]
  We introduce a novel approach to directly optimize a reinforcement learning objective, maximizing an expected reward. We test our methodology on two tasks: generating molecules with user-defined properties and identifying short python expressions which evaluate to a given target value.
  arXiv  Detail & Related papers  (2020-10-05T20:03:13Z)
• Data Transfer Approaches to Improve Seq-to-Seq Retrosynthesis [1.6449390849183363]
  Retrosynthesis is a problem to infer reactant compounds to synthesize a given product compound through chemical reactions. Recent studies on retrosynthesis focus on proposing more sophisticated prediction models. The dataset to feed the models also plays an essential role in achieving the best generalizing models.
  arXiv  Detail & Related papers  (2020-10-02T05:27:51Z)
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study [81.11161697133095]
  We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models. As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
  arXiv  Detail & Related papers  (2020-01-12T04:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.