Data-Free Adversarial Knowledge Distillation for Graph Neural Networks

• URL: http://arxiv.org/abs/2205.03811v1
• Date: Sun, 8 May 2022 08:19:40 GMT
• Title: Data-Free Adversarial Knowledge Distillation for Graph Neural Networks
• Authors: Yuanxin Zhuang, Lingjuan Lyu, Chuan Shi, Carl Yang, Lichao Sun
• Abstract summary: We propose the first end-to-end framework for data-free adversarial knowledge distillation on graph structured data (DFAD-GNN) To be specific, our DFAD-GNN employs a generative adversarial network, which mainly consists of three components: a pre-trained teacher model and a student model are regarded as two discriminators, and a generator is utilized for deriving training graphs to distill knowledge from the teacher model into the student model. Our DFAD-GNN significantly surpasses state-of-the-art data-free baselines in the graph classification task.
• Score: 62.71646916191515
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Graph neural networks (GNNs) have been widely used in modeling graph structured data, owing to its impressive performance in a wide range of practical applications. Recently, knowledge distillation (KD) for GNNs has enabled remarkable progress in graph model compression and knowledge transfer. However, most of the existing KD methods require a large volume of real data, which are not readily available in practice, and may preclude their applicability in scenarios where the teacher model is trained on rare or hard to acquire datasets. To address this problem, we propose the first end-to-end framework for data-free adversarial knowledge distillation on graph structured data (DFAD-GNN). To be specific, our DFAD-GNN employs a generative adversarial network, which mainly consists of three components: a pre-trained teacher model and a student model are regarded as two discriminators, and a generator is utilized for deriving training graphs to distill knowledge from the teacher model into the student model. Extensive experiments on various benchmark models and six representative datasets demonstrate that our DFAD-GNN significantly surpasses state-of-the-art data-free baselines in the graph classification task.

Related papers

• GOODAT: Towards Test-time Graph Out-of-Distribution Detection [103.40396427724667]
  Graph neural networks (GNNs) have found widespread application in modeling graph data across diverse domains. Recent studies have explored graph OOD detection, often focusing on training a specific model or modifying the data on top of a well-trained GNN. This paper introduces a data-centric, unsupervised, and plug-and-play solution that operates independently of training data and modifications of GNN architecture.
  arXiv  Detail & Related papers  (2024-01-10T08:37:39Z)
• Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis [50.972595036856035]
  We present a code that successfully replicates results from six popular and recent graph recommendation models. We compare these graph models with traditional collaborative filtering models that historically performed well in offline evaluations. By investigating the information flow from users' neighborhoods, we aim to identify which models are influenced by intrinsic features in the dataset structure.
  arXiv  Detail & Related papers  (2023-08-01T09:31:44Z)
• Frameless Graph Knowledge Distillation [27.831929635701886]
  We show how the graph knowledge supplied by the teacher is learned and digested by the student model via both algebra and geometry. Our proposed model can generate learning accuracy identical to or even surpass the teacher model while maintaining the high speed of inference.
  arXiv  Detail & Related papers  (2023-07-13T08:56:50Z)
• Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge Distillation [65.62538699160085]
  We propose a Directed Acyclic Graph Factorization Machine (KD-DAGFM) to learn the high-order feature interactions from existing complex interaction models for CTR prediction via Knowledge Distillation. KD-DAGFM achieves the best performance with less than 21.5% FLOPs of the state-of-the-art method on both online and offline experiments.
  arXiv  Detail & Related papers  (2022-11-21T03:09:42Z)
• EXPERT: Public Benchmarks for Dynamic Heterogeneous Academic Graphs [5.4744970832051445]
  We present a variety of large scale, dynamic heterogeneous academic graphs to test the effectiveness of models developed for graph forecasting tasks. Our novel datasets cover both context and content information extracted from scientific publications across two communities: Artificial Intelligence (AI) and Nuclear Nonproliferation (NN)
  arXiv  Detail & Related papers  (2022-04-14T19:43:34Z)
• Multi network InfoMax: A pre-training method involving graph convolutional networks [0.0]
  This paper presents a pre-training method involving graph convolutional/neural networks (GCNs/GNNs) The learned high-level graph latent representations help increase performance for downstream graph classification tasks. We apply our method to a neuroimaging dataset for classifying subjects into healthy control (HC) and schizophrenia (SZ) groups.
  arXiv  Detail & Related papers  (2021-11-01T21:53:20Z)
• Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework [42.57467126227328]
  We propose a framework based on knowledge distillation to address the issues of semi-supervised learning on graphs. Our framework extracts the knowledge of an arbitrary learned GNN model (teacher model) and injects it into a well-designed student model. Experimental results show that the learned student model can consistently outperform its corresponding teacher model by 1.4% - 4.7% on average.
  arXiv  Detail & Related papers  (2021-03-04T08:13:55Z)
• GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training [62.73470368851127]
  Graph representation learning has emerged as a powerful technique for addressing real-world problems. We design Graph Contrastive Coding -- a self-supervised graph neural network pre-training framework. We conduct experiments on three graph learning tasks and ten graph datasets.
  arXiv  Detail & Related papers  (2020-06-17T16:18:35Z)
• Self-supervised Training of Graph Convolutional Networks [39.80867112204255]
  Graph Convolutional Networks (GCNs) have been successfully applied to analyze non-grid data. In this paper, we propose two types of self-supervised learning strategies to exploit available information from the input graph structure data itself.
  arXiv  Detail & Related papers  (2020-06-03T16:53:37Z)
• Distilling Knowledge from Graph Convolutional Networks [146.71503336770886]
  Existing knowledge distillation methods focus on convolutional neural networks (CNNs) We propose the first dedicated approach to distilling knowledge from a pre-trained graph convolutional network (GCN) model. We show that our method achieves the state-of-the-art knowledge distillation performance for GCN models.
  arXiv  Detail & Related papers  (2020-03-23T18:23:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.