Benchmarking Accuracy and Generalizability of Four Graph Neural Networks
Using Large In Vitro ADME Datasets from Different Chemical Spaces
- URL: http://arxiv.org/abs/2111.13964v1
- Date: Sat, 27 Nov 2021 18:54:38 GMT
- Title: Benchmarking Accuracy and Generalizability of Four Graph Neural Networks
Using Large In Vitro ADME Datasets from Different Chemical Spaces
- Authors: Fabio Broccatelli, Richard Trager, Michael Reutlinger, George Karypis,
Mufei Li
- Abstract summary: We consider four graph neural network (GNN) variants -- Graph Convolutional Network (GCN), Graph Attention Network (GAT), Message Passing Neural Network (MPNN) and Attentive Fingerprint (AttentiveFP)
All GNN models significantly outperform lower-bar benchmark traditional models solely based on fingerprints.
Only GATs seem to offer a small but consistent improvement over higher-bar benchmark traditional models.
- Score: 6.118940071203314
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this work, we benchmark a variety of single- and multi-task graph neural
network (GNN) models against lower-bar and higher-bar traditional machine
learning approaches employing human engineered molecular features. We consider
four GNN variants -- Graph Convolutional Network (GCN), Graph Attention Network
(GAT), Message Passing Neural Network (MPNN), and Attentive Fingerprint
(AttentiveFP). So far deep learning models have been primarily benchmarked
using lower-bar traditional models solely based on fingerprints, while more
realistic benchmarks employing fingerprints, whole-molecule descriptors and
predictions from other related endpoints (e.g., LogD7.4) appear to be scarce
for industrial ADME datasets. In addition to time-split test sets based on
Genentech data, this study benefits from the availability of measurements from
an external chemical space (Roche data). We identify GAT as a promising
approach to implementing deep learning models. While all GNN models
significantly outperform lower-bar benchmark traditional models solely based on
fingerprints, only GATs seem to offer a small but consistent improvement over
higher-bar benchmark traditional models. Finally, the accuracy of in vitro
assays from different laboratories predicting the same experimental endpoints
appears to be comparable with the accuracy of GAT single-task models,
suggesting that most of the observed error from the models is a function of the
experimental error propagation.
Related papers
- GOODAT: Towards Test-time Graph Out-of-Distribution Detection [103.40396427724667]
Graph neural networks (GNNs) have found widespread application in modeling graph data across diverse domains.
Recent studies have explored graph OOD detection, often focusing on training a specific model or modifying the data on top of a well-trained GNN.
This paper introduces a data-centric, unsupervised, and plug-and-play solution that operates independently of training data and modifications of GNN architecture.
arXiv Detail & Related papers (2024-01-10T08:37:39Z) - Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis [50.972595036856035]
We present a code that successfully replicates results from six popular and recent graph recommendation models.
We compare these graph models with traditional collaborative filtering models that historically performed well in offline evaluations.
By investigating the information flow from users' neighborhoods, we aim to identify which models are influenced by intrinsic features in the dataset structure.
arXiv Detail & Related papers (2023-08-01T09:31:44Z) - Data-Free Adversarial Knowledge Distillation for Graph Neural Networks [62.71646916191515]
We propose the first end-to-end framework for data-free adversarial knowledge distillation on graph structured data (DFAD-GNN)
To be specific, our DFAD-GNN employs a generative adversarial network, which mainly consists of three components: a pre-trained teacher model and a student model are regarded as two discriminators, and a generator is utilized for deriving training graphs to distill knowledge from the teacher model into the student model.
Our DFAD-GNN significantly surpasses state-of-the-art data-free baselines in the graph classification task.
arXiv Detail & Related papers (2022-05-08T08:19:40Z) - Characterizing and Understanding the Behavior of Quantized Models for
Reliable Deployment [32.01355605506855]
Quantization-aware training can produce more stable models than standard, adversarial, and Mixup training.
Disagreements often have closer top-1 and top-2 output probabilities, and $Margin$ is a better indicator than the other uncertainty metrics to distinguish disagreements.
We opensource our code and models as a new benchmark for further studying the quantized models.
arXiv Detail & Related papers (2022-04-08T11:19:16Z) - Graph Neural Networks with Parallel Neighborhood Aggregations for Graph
Classification [14.112444998191698]
We focus on graph classification using a graph neural network (GNN) model that precomputes the node features using a bank of neighborhood aggregation graph operators arranged in parallel.
These GNN models have a natural advantage of reduced training and inference time due to the precomputations.
We demonstrate via numerical experiments that the developed model achieves state-of-the-art performance on many diverse real-world datasets.
arXiv Detail & Related papers (2021-11-22T19:19:40Z) - A Comparative Study on Basic Elements of Deep Learning Models for
Spatial-Temporal Traffic Forecasting [0.0]
Traffic forecasting plays a crucial role in intelligent transportation systems.
The recently suggested deep learning models share basic elements such as graph convolution, graph attention, recurrent units, and/or attention mechanism.
In this study, we designed an in-depth comparative study for four deep neural network models utilizing different basic elements.
arXiv Detail & Related papers (2021-11-15T03:20:23Z) - NODE-GAM: Neural Generalized Additive Model for Interpretable Deep
Learning [16.15084484295732]
Generalized Additive Models (GAMs) have a long history of use in high-risk domains.
We propose a neural GAM (NODE-GAM) and neural GA$2$M (NODE-GA$2$M)
We show that our proposed models have comparable accuracy to other non-interpretable models, and outperform other GAMs on large datasets.
arXiv Detail & Related papers (2021-06-03T06:20:18Z) - ALT-MAS: A Data-Efficient Framework for Active Testing of Machine
Learning Algorithms [58.684954492439424]
We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data.
The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
arXiv Detail & Related papers (2021-04-11T12:14:04Z) - Firearm Detection via Convolutional Neural Networks: Comparing a
Semantic Segmentation Model Against End-to-End Solutions [68.8204255655161]
Threat detection of weapons and aggressive behavior from live video can be used for rapid detection and prevention of potentially deadly incidents.
One way for achieving this is through the use of artificial intelligence and, in particular, machine learning for image analysis.
We compare a traditional monolithic end-to-end deep learning model and a previously proposed model based on an ensemble of simpler neural networks detecting fire-weapons via semantic segmentation.
arXiv Detail & Related papers (2020-12-17T15:19:29Z) - Combining Label Propagation and Simple Models Out-performs Graph Neural
Networks [52.121819834353865]
We show that for many standard transductive node classification benchmarks, we can exceed or match the performance of state-of-the-art GNNs.
We call this overall procedure Correct and Smooth (C&S)
Our approach exceeds or nearly matches the performance of state-of-the-art GNNs on a wide variety of benchmarks.
arXiv Detail & Related papers (2020-10-27T02:10:52Z) - Ensembles of Spiking Neural Networks [0.3007949058551534]
This paper demonstrates how to construct ensembles of spiking neural networks producing state-of-the-art results.
We achieve classification accuracies of 98.71%, 100.0%, and 99.09%, on the MNIST, NMNIST and DVS Gesture datasets respectively.
We formalize spiking neural networks as GLM predictors, identifying a suitable representation for their target domain.
arXiv Detail & Related papers (2020-10-15T17:45:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.