Benchmarking Accuracy and Generalizability of Four Graph Neural Networks Using Large In Vitro ADME Datasets from Different Chemical Spaces

• URL: http://arxiv.org/abs/2111.13964v1
• Date: Sat, 27 Nov 2021 18:54:38 GMT
• Title: Benchmarking Accuracy and Generalizability of Four Graph Neural Networks Using Large In Vitro ADME Datasets from Different Chemical Spaces
• Authors: Fabio Broccatelli, Richard Trager, Michael Reutlinger, George Karypis, Mufei Li
• Abstract summary: We consider four graph neural network (GNN) variants -- Graph Convolutional Network (GCN), Graph Attention Network (GAT), Message Passing Neural Network (MPNN) and Attentive Fingerprint (AttentiveFP) All GNN models significantly outperform lower-bar benchmark traditional models solely based on fingerprints. Only GATs seem to offer a small but consistent improvement over higher-bar benchmark traditional models.
• Score: 6.118940071203314
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we benchmark a variety of single- and multi-task graph neural network (GNN) models against lower-bar and higher-bar traditional machine learning approaches employing human engineered molecular features. We consider four GNN variants -- Graph Convolutional Network (GCN), Graph Attention Network (GAT), Message Passing Neural Network (MPNN), and Attentive Fingerprint (AttentiveFP). So far deep learning models have been primarily benchmarked using lower-bar traditional models solely based on fingerprints, while more realistic benchmarks employing fingerprints, whole-molecule descriptors and predictions from other related endpoints (e.g., LogD7.4) appear to be scarce for industrial ADME datasets. In addition to time-split test sets based on Genentech data, this study benefits from the availability of measurements from an external chemical space (Roche data). We identify GAT as a promising approach to implementing deep learning models. While all GNN models significantly outperform lower-bar benchmark traditional models solely based on fingerprints, only GATs seem to offer a small but consistent improvement over higher-bar benchmark traditional models. Finally, the accuracy of in vitro assays from different laboratories predicting the same experimental endpoints appears to be comparable with the accuracy of GAT single-task models, suggesting that most of the observed error from the models is a function of the experimental error propagation.

Related papers

• PROXI: Challenging the GNNs for Link Prediction [3.8233569758620063]
  We introduce PROXI, which leverages proximity information of node pairs in both graph and attribute spaces. Standard machine learning (ML) models perform competitively, even outperforming cutting-edge GNN models. We show that augmenting traditional GNNs with PROXI significantly boosts their link prediction performance.
  arXiv  Detail & Related papers  (2024-10-02T17:57:38Z)
• Generalization of Graph Neural Networks is Robust to Model Mismatch [84.01980526069075]
  Graph neural networks (GNNs) have demonstrated their effectiveness in various tasks supported by their generalization capabilities. In this paper, we examine GNNs that operate on geometric graphs generated from manifold models. Our analysis reveals the robustness of the GNN generalization in the presence of such model mismatch.
  arXiv  Detail & Related papers  (2024-08-25T16:00:44Z)
• Variational autoencoder-based neural network model compression [4.992476489874941]
  Variational Autoencoders (VAEs), as a form of deep generative model, have been widely used in recent years. This paper aims to explore neural network model compression method based on VAE.
  arXiv  Detail & Related papers  (2024-08-25T09:06:22Z)
• GOODAT: Towards Test-time Graph Out-of-Distribution Detection [103.40396427724667]
  Graph neural networks (GNNs) have found widespread application in modeling graph data across diverse domains. Recent studies have explored graph OOD detection, often focusing on training a specific model or modifying the data on top of a well-trained GNN. This paper introduces a data-centric, unsupervised, and plug-and-play solution that operates independently of training data and modifications of GNN architecture.
  arXiv  Detail & Related papers  (2024-01-10T08:37:39Z)
• Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis [50.972595036856035]
  We present a code that successfully replicates results from six popular and recent graph recommendation models. We compare these graph models with traditional collaborative filtering models that historically performed well in offline evaluations. By investigating the information flow from users' neighborhoods, we aim to identify which models are influenced by intrinsic features in the dataset structure.
  arXiv  Detail & Related papers  (2023-08-01T09:31:44Z)
• Characterizing and Understanding the Behavior of Quantized Models for Reliable Deployment [32.01355605506855]
  Quantization-aware training can produce more stable models than standard, adversarial, and Mixup training. Disagreements often have closer top-1 and top-2 output probabilities, and $Margin$ is a better indicator than the other uncertainty metrics to distinguish disagreements. We opensource our code and models as a new benchmark for further studying the quantized models.
  arXiv  Detail & Related papers  (2022-04-08T11:19:16Z)
• Graph Neural Networks with Parallel Neighborhood Aggregations for Graph Classification [14.112444998191698]
  We focus on graph classification using a graph neural network (GNN) model that precomputes the node features using a bank of neighborhood aggregation graph operators arranged in parallel. These GNN models have a natural advantage of reduced training and inference time due to the precomputations. We demonstrate via numerical experiments that the developed model achieves state-of-the-art performance on many diverse real-world datasets.
  arXiv  Detail & Related papers  (2021-11-22T19:19:40Z)
• NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning [16.15084484295732]
  Generalized Additive Models (GAMs) have a long history of use in high-risk domains. We propose a neural GAM (NODE-GAM) and neural GA$2$M (NODE-GA$2$M) We show that our proposed models have comparable accuracy to other non-interpretable models, and outperform other GAMs on large datasets.
  arXiv  Detail & Related papers  (2021-06-03T06:20:18Z)
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms [58.684954492439424]
  We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
  arXiv  Detail & Related papers  (2021-04-11T12:14:04Z)
• Firearm Detection via Convolutional Neural Networks: Comparing a Semantic Segmentation Model Against End-to-End Solutions [68.8204255655161]
  Threat detection of weapons and aggressive behavior from live video can be used for rapid detection and prevention of potentially deadly incidents. One way for achieving this is through the use of artificial intelligence and, in particular, machine learning for image analysis. We compare a traditional monolithic end-to-end deep learning model and a previously proposed model based on an ensemble of simpler neural networks detecting fire-weapons via semantic segmentation.
  arXiv  Detail & Related papers  (2020-12-17T15:19:29Z)
• Combining Label Propagation and Simple Models Out-performs Graph Neural Networks [52.121819834353865]
  We show that for many standard transductive node classification benchmarks, we can exceed or match the performance of state-of-the-art GNNs. We call this overall procedure Correct and Smooth (C&S) Our approach exceeds or nearly matches the performance of state-of-the-art GNNs on a wide variety of benchmarks.
  arXiv  Detail & Related papers  (2020-10-27T02:10:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.