Pushing the Limits of All-Atom Geometric Graph Neural Networks: Pre-Training, Scaling and Zero-Shot Transfer

• URL: http://arxiv.org/abs/2410.21683v1
• Date: Tue, 29 Oct 2024 03:07:33 GMT
• Title: Pushing the Limits of All-Atom Geometric Graph Neural Networks: Pre-Training, Scaling and Zero-Shot Transfer
• Authors: Zihan Pengmei, Zhengyuan Shen, Zichen Wang, Marcus Collins, Huzefa Rangwala,
• Abstract summary: Geometric graph neural networks (Geom-GNNs) with all-atom information have transformed atomistic simulations. We study the scaling behaviors of Geom-GNNs under self-supervised pre-training, supervised and unsupervised learning setups. We show how all-atom graph embedding can be organically combined with other neural architectures to enhance the expressive power.
• Score: 15.302727191576784
• License:
• Abstract: Constructing transferable descriptors for conformation representation of molecular and biological systems finds numerous applications in drug discovery, learning-based molecular dynamics, and protein mechanism analysis. Geometric graph neural networks (Geom-GNNs) with all-atom information have transformed atomistic simulations by serving as a general learnable geometric descriptors for downstream tasks including prediction of interatomic potential and molecular properties. However, common practices involve supervising Geom-GNNs on specific downstream tasks, which suffer from the lack of high-quality data and inaccurate labels leading to poor generalization and performance degradation on out-of-distribution (OOD) scenarios. In this work, we explored the possibility of using pre-trained Geom-GNNs as transferable and highly effective geometric descriptors for improved generalization. To explore their representation power, we studied the scaling behaviors of Geom-GNNs under self-supervised pre-training, supervised and unsupervised learning setups. We find that the expressive power of different architectures can differ on the pre-training task. Interestingly, Geom-GNNs do not follow the power-law scaling on the pre-training task, and universally lack predictable scaling behavior on the supervised tasks with quantum chemical labels important for screening and design of novel molecules. More importantly, we demonstrate how all-atom graph embedding can be organically combined with other neural architectures to enhance the expressive power. Meanwhile, the low-dimensional projection of the latent space shows excellent agreement with conventional geometrical descriptors.

Related papers

• Graph Neural Networks Uncover Geometric Neural Representations in Reinforcement-Based Motor Learning [3.379988469252273]
  Graph Neural Networks (GNN) can capture the geometric properties of neural representations in EEG data. We study how reinforcement-based motor learning affects neural activity patterns during motor planning.
  arXiv  Detail & Related papers  (2024-10-31T10:54:50Z)
• geom2vec: pretrained GNNs as geometric featurizers for conformational dynamics [0.0]
  We introduce geom2vec, in which pretrained graph neural networks (GNNs) are used as universal featurizers. We learn transferable structural representations that capture molecular geometric patterns without further fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T00:36:06Z)
• Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks [80.8446673089281]
  We study a new paradigm of knowledge transfer that aims at encoding graph topological information into graph neural networks (GNNs) We propose Neural Heat Kernel (NHK) to encapsulate the geometric property of the underlying manifold concerning the architecture of GNNs. A fundamental and principled solution is derived by aligning NHKs on teacher and student models, dubbed as Geometric Knowledge Distillation.
  arXiv  Detail & Related papers  (2022-10-24T08:01:58Z)
• MentorGNN: Deriving Curriculum for Pre-Training GNNs [61.97574489259085]
  We propose an end-to-end model named MentorGNN that aims to supervise the pre-training process of GNNs across graphs. We shed new light on the problem of domain adaption on relational data (i.e., graphs) by deriving a natural and interpretable upper bound on the generalization error of the pre-trained GNNs.
  arXiv  Detail & Related papers  (2022-08-21T15:12:08Z)
• Graph neural networks for the prediction of molecular structure-property relationships [59.11160990637615]
  Graph neural networks (GNNs) are a novel machine learning method that directly work on the molecular graph. GNNs allow to learn properties in an end-to-end fashion, thereby avoiding the need for informative descriptors. We describe the fundamentals of GNNs and demonstrate the application of GNNs via two examples for molecular property prediction.
  arXiv  Detail & Related papers  (2022-07-25T11:30:44Z)
• Simple and Efficient Heterogeneous Graph Neural Network [55.56564522532328]
  Heterogeneous graph neural networks (HGNNs) have powerful capability to embed rich structural and semantic information of a heterogeneous graph into node representations. Existing HGNNs inherit many mechanisms from graph neural networks (GNNs) over homogeneous graphs, especially the attention mechanism and the multi-layer structure. This paper conducts an in-depth and detailed study of these mechanisms and proposes Simple and Efficient Heterogeneous Graph Neural Network (SeHGNN)
  arXiv  Detail & Related papers  (2022-07-06T10:01:46Z)
• Spiking Graph Convolutional Networks [19.36064180392385]
  SpikingGCN is an end-to-end framework that aims to integrate the embedding of GCNs with the biofidelity characteristics of SNNs. We show that SpikingGCN on a neuromorphic chip can bring a clear advantage of energy efficiency into graph data analysis.
  arXiv  Detail & Related papers  (2022-05-05T16:44:36Z)
• Hyperbolic Graph Neural Networks: A Review of Methods and Applications [55.5502008501764]
  Graph neural networks generalize conventional neural networks to graph-structured data. The performance of Euclidean models in graph-related learning is still bounded and limited by the representation ability of Euclidean geometry. Recently, hyperbolic space has gained increasing popularity in processing graph data with tree-like structure and power-law distribution.
  arXiv  Detail & Related papers  (2022-02-28T15:08:48Z)
• GeomGCL: Geometric Graph Contrastive Learning for Molecular Property Prediction [47.70253904390288]
  We propose a novel graph contrastive learning method utilizing the geometry of a molecule across 2D and 3D views. Specifically, we first devise a dual-view geometric message passing network (GeomMPNN) to adaptively leverage the rich information of both 2D and 3D graphs of a molecule.
  arXiv  Detail & Related papers  (2021-09-24T03:55:27Z)
• Exploiting Spiking Dynamics with Spatial-temporal Feature Normalization in Graph Learning [9.88508686848173]
  Biological spiking neurons with intrinsic dynamics underlie the powerful representation and learning capabilities of the brain. Despite recent tremendous progress in spiking neural networks (SNNs) for handling Euclidean-space tasks, it still remains challenging to exploit SNNs in processing non-Euclidean-space data. Here we present a general spike-based modeling framework that enables the direct training of SNNs for graph learning.
  arXiv  Detail & Related papers  (2021-06-30T11:20:16Z)
• Distance-aware Molecule Graph Attention Network for Drug-Target Binding Affinity Prediction [54.93890176891602]
  We propose a diStance-aware Molecule graph Attention Network (S-MAN) tailored to drug-target binding affinity prediction. As a dedicated solution, we first propose a position encoding mechanism to integrate the topological structure and spatial position information into the constructed pocket-ligand graph. We also propose a novel edge-node hierarchical attentive aggregation structure which has edge-level aggregation and node-level aggregation.
  arXiv  Detail & Related papers  (2020-12-17T17:44:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.