How Do Graph Networks Generalize to Large and Diverse Molecular Systems?

• URL: http://arxiv.org/abs/2204.02782v1
• Date: Wed, 6 Apr 2022 12:52:34 GMT
• Title: How Do Graph Networks Generalize to Large and Diverse Molecular Systems?
• Authors: Johannes Gasteiger, Muhammed Shuaibi, Anuroop Sriram, Stephan G\"unnemann, Zachary Ulissi, C. Lawrence Zitnick, Abhishek Das
• Abstract summary: We identify four aspects of complexity in which many datasets are lacking. We propose the GemNet-OC model, which outperforms the previous state-of-the-art on OC20 by 16%. Our findings challenge the common belief that graph neural networks work equally well independent of dataset size and diversity.
• Score: 10.690849483282564
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The predominant method of demonstrating progress of atomic graph neural networks are benchmarks on small and limited datasets. The implicit hypothesis behind this approach is that progress on these narrow datasets generalize to the large diversity of chemistry. This generalizability would be very helpful for research, but currently remains untested. In this work we test this assumption by identifying four aspects of complexity in which many datasets are lacking: 1. Chemical diversity (number of different elements), 2. system size (number of atoms per sample), 3. dataset size (number of data samples), and 4. domain shift (similarity of the training and test set). We introduce multiple subsets of the large Open Catalyst 2020 (OC20) dataset to independently investigate each of these aspects. We then perform 21 ablation studies and sensitivity analyses on 9 datasets testing both previously proposed and new model enhancements. We find that some improvements are consistent between datasets, but many are not and some even have opposite effects. Based on this analysis, we identify a smaller dataset that correlates well with the full OC20 dataset, and propose the GemNet-OC model, which outperforms the previous state-of-the-art on OC20 by 16%, while reducing training time by a factor of 10. Overall, our findings challenge the common belief that graph neural networks work equally well independent of dataset size and diversity, and suggest that caution must be exercised when making generalizations based on narrow datasets.

Related papers

• UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction [93.77809355002591]
  We introduce UniTraj, a comprehensive framework that unifies various datasets, models, and evaluation criteria. We conduct extensive experiments and find that model performance significantly drops when transferred to other datasets. We provide insights into dataset characteristics to explain these findings.
  arXiv  Detail & Related papers  (2024-03-22T10:36:50Z)
• DaFoEs: Mixing Datasets towards the generalization of vision-state deep-learning Force Estimation in Minimally Invasive Robotic Surgery [6.55111164866752]
  We present a new vision-haptic dataset (DaFoEs) with variable soft environments for the training of deep neural models. We also present a variable encoder-decoder architecture to predict the forces done by the laparoscopic tool using single input or sequence of inputs.
  arXiv  Detail & Related papers  (2024-01-17T14:39:55Z)
• Replication: Contrastive Learning and Data Augmentation in Traffic Classification Using a Flowpic Input Representation [47.95762911696397]
  We reproduce [16] on the same datasets and replicate its most salient aspect (the importance of data augmentation) on three additional public datasets. While we confirm most of the original results, we also found a 20% accuracy drop on some of the investigated scenarios due to a data shift in the original dataset.
  arXiv  Detail & Related papers  (2023-09-18T12:55:09Z)
• Kernel Regression with Infinite-Width Neural Networks on Millions of Examples [27.408712993696213]
  We study scaling laws of several neural kernels across many orders of magnitude for the CIFAR-5m dataset. We obtain a test accuracy of 91.2% (SotA for a pure kernel method)
  arXiv  Detail & Related papers  (2023-03-09T17:11:31Z)
• Ensemble Machine Learning Model Trained on a New Synthesized Dataset Generalizes Well for Stress Prediction Using Wearable Devices [3.006016887654771]
  We investigate the generalization ability of models built on datasets containing a small number of subjects, recorded in single study protocols. We propose and evaluate the use of ensemble techniques by combining gradient boosting with an artificial neural network to measure predictive power on new, unseen data.
  arXiv  Detail & Related papers  (2022-09-30T00:20:57Z)
• Condensing Graphs via One-Step Gradient Matching [50.07587238142548]
  We propose a one-step gradient matching scheme, which performs gradient matching for only one single step without training the network weights. Our theoretical analysis shows this strategy can generate synthetic graphs that lead to lower classification loss on real graphs. In particular, we are able to reduce the dataset size by 90% while approximating up to 98% of the original performance.
  arXiv  Detail & Related papers  (2022-06-15T18:20:01Z)
• Efficient Analysis of COVID-19 Clinical Data using Machine Learning Models [0.0]
  Huge volumes of data and case studies have been made available, providing researchers with a unique opportunity to find trends. Applying machine learning based algorithms to this big data is a natural approach to take to this aim. We show that with the efficient feature selection algorithm, we can achieve a prediction accuracy of more than 90% in most cases.
  arXiv  Detail & Related papers  (2021-10-18T20:06:01Z)
• Networked Time Series Prediction with Incomplete Data [59.45358694862176]
  We propose NETS-ImpGAN, a novel deep learning framework that can be trained on incomplete data with missing values in both history and future. We conduct extensive experiments on three real-world datasets under different missing patterns and missing rates.
  arXiv  Detail & Related papers  (2021-10-05T18:20:42Z)
• Comparing Test Sets with Item Response Theory [53.755064720563]
  We evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models. We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models.
  arXiv  Detail & Related papers  (2021-06-01T22:33:53Z)
• Modeling Shared Responses in Neuroimaging Studies through MultiView ICA [94.31804763196116]
  Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization. We propose a novel MultiView Independent Component Analysis model for group studies, where data from each subject are modeled as a linear combination of shared independent sources plus noise. We demonstrate the usefulness of our approach first on fMRI data, where our model demonstrates improved sensitivity in identifying common sources among subjects.
  arXiv  Detail & Related papers  (2020-06-11T17:29:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.