Accelerating Molecular Graph Neural Networks via Knowledge Distillation

• URL: http://arxiv.org/abs/2306.14818v2
• Date: Sat, 28 Oct 2023 10:55:37 GMT
• Title: Accelerating Molecular Graph Neural Networks via Knowledge Distillation
• Authors: Filip Ekstr\"om Kelvinius, Dimitar Georgiev, Artur Petrov Toshev, Johannes Gasteiger
• Abstract summary: Recent advances in graph neural networks (GNNs) have enabled more comprehensive modeling of molecules and molecular systems. As the field has been progressing to bigger and more complex architectures, state-of-the-art GNNs have become largely prohibitive for many large-scale applications. We devise KD strategies that facilitate the distillation of hidden representations in directional and equivariant GNNs, and evaluate their performance on the regression task of energy and force prediction.
• Score: 1.9116784879310031
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in graph neural networks (GNNs) have enabled more comprehensive modeling of molecules and molecular systems, thereby enhancing the precision of molecular property prediction and molecular simulations. Nonetheless, as the field has been progressing to bigger and more complex architectures, state-of-the-art GNNs have become largely prohibitive for many large-scale applications. In this paper, we explore the utility of knowledge distillation (KD) for accelerating molecular GNNs. To this end, we devise KD strategies that facilitate the distillation of hidden representations in directional and equivariant GNNs, and evaluate their performance on the regression task of energy and force prediction. We validate our protocols across different teacher-student configurations and datasets, and demonstrate that they can consistently boost the predictive accuracy of student models without any modifications to their architecture. Moreover, we conduct comprehensive optimization of various components of our framework, and investigate the potential of data augmentation to further enhance performance. All in all, we manage to close the gap in predictive accuracy between teacher and student models by as much as 96.7% and 62.5% for energy and force prediction respectively, while fully preserving the inference throughput of the more lightweight models.

Related papers

• Dumpling GNN: Hybrid GNN Enables Better ADC Payload Activity Prediction Based on Chemical Structure [53.76752789814785]
  DumplingGNN is a hybrid Graph Neural Network architecture specifically designed for predicting ADC payload activity based on chemical structure. We evaluate it on a comprehensive ADC payload dataset focusing on DNA Topoisomerase I inhibitors. It demonstrates exceptional accuracy (91.48%), sensitivity (95.08%), and specificity (97.54%) on our specialized ADC payload dataset.
  arXiv  Detail & Related papers  (2024-09-23T17:11:04Z)
• Cross-Modal Learning for Chemistry Property Prediction: Large Language Models Meet Graph Machine Learning [0.0]
  We introduce a Multi-Modal Fusion (MMF) framework that harnesses the analytical prowess of Graph Neural Networks (GNNs) and the linguistic generative and predictive abilities of Large Language Models (LLMs) Our framework combines the effectiveness of GNNs in modeling graph-structured data with the zero-shot and few-shot learning capabilities of LLMs, enabling improved predictions while reducing the risk of overfitting.
  arXiv  Detail & Related papers  (2024-08-27T11:10:39Z)
• GNN-SKAN: Harnessing the Power of SwallowKAN to Advance Molecular Representation Learning with GNNs [19.019980841275366]
  We introduce a new class of GNNs that integrates the Kolmogorov-Arnold Networks (KANs) KANs are known for their robust data-fitting capabilities and high accuracy in small-scale AI + Science tasks. We propose a new class of GNNs, GNN-SKAN, and its augmented variant, GNN-SKAN+, which incorporates a SKAN-based classifier to further boost performance.
  arXiv  Detail & Related papers  (2024-08-02T05:36:14Z)
• Bi-level Contrastive Learning for Knowledge-Enhanced Molecule Representations [55.42602325017405]
  We propose a novel method called GODE, which takes into account the two-level structure of individual molecules. By pre-training two graph neural networks (GNNs) on different graph structures, combined with contrastive learning, GODE fuses molecular structures with their corresponding knowledge graph substructures. When fine-tuned across 11 chemical property tasks, our model outperforms existing benchmarks, registering an average ROC-AUC uplift of 13.8% for classification tasks and an average RMSE/MAE enhancement of 35.1% for regression tasks.
  arXiv  Detail & Related papers  (2023-06-02T15:49:45Z)
• Molecule-Morphology Contrastive Pretraining for Transferable Molecular Representation [0.0]
  We introduce Molecule-Morphology Contrastive Pretraining (MoCoP), a framework for learning multi-modal representation of molecular graphs and cellular morphologies. We scale MoCoP to approximately 100K molecules and 600K morphological profiles using data from the JUMP-CP Consortium. Our findings suggest that integrating cellular morphologies with molecular graphs using MoCoP can significantly improve the performance of QSAR models.
  arXiv  Detail & Related papers  (2023-04-27T02:01:41Z)
• Graph neural networks for the prediction of molecular structure-property relationships [59.11160990637615]
  Graph neural networks (GNNs) are a novel machine learning method that directly work on the molecular graph. GNNs allow to learn properties in an end-to-end fashion, thereby avoiding the need for informative descriptors. We describe the fundamentals of GNNs and demonstrate the application of GNNs via two examples for molecular property prediction.
  arXiv  Detail & Related papers  (2022-07-25T11:30:44Z)
• KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction [13.55018269009361]
  We introduce Knowledge-guided Pre-training of Graph Transformer (KPGT), a novel self-supervised learning framework for molecular graph representation learning. KPGT can offer superior performance over current state-of-the-art methods on several molecular property prediction tasks.
  arXiv  Detail & Related papers  (2022-06-02T08:22:14Z)
• Few-Shot Graph Learning for Molecular Property Prediction [46.60746023179724]
  We propose Meta-MGNN, a novel model for few-shot molecular property prediction. To exploit unlabeled molecular information, Meta-MGNN further incorporates molecular structure, attribute based self-supervised modules and self-attentive task weights. Extensive experiments on two public multi-property datasets demonstrate that Meta-MGNN outperforms a variety of state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-02-16T01:55:34Z)
• Self-Supervised Graph Transformer on Large-Scale Molecular Data [73.3448373618865]
  We propose a novel framework, GROVER, for molecular representation learning. GROVER can learn rich structural and semantic information of molecules from enormous unlabelled molecular data. We pre-train GROVER with 100 million parameters on 10 million unlabelled molecules -- the biggest GNN and the largest training dataset in molecular representation learning.
  arXiv  Detail & Related papers  (2020-06-18T08:37:04Z)
• Multi-View Graph Neural Networks for Molecular Property Prediction [67.54644592806876]
  We present Multi-View Graph Neural Network (MV-GNN), a multi-view message passing architecture. In MV-GNN, we introduce a shared self-attentive readout component and disagreement loss to stabilize the training process. We further boost the expressive power of MV-GNN by proposing a cross-dependent message passing scheme.
  arXiv  Detail & Related papers  (2020-05-17T04:46:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.