Addressing Over-Smoothing in Graph Neural Networks via Deep Supervision

• URL: http://arxiv.org/abs/2202.12508v1
• Date: Fri, 25 Feb 2022 06:05:55 GMT
• Title: Addressing Over-Smoothing in Graph Neural Networks via Deep Supervision
• Authors: Pantelis Elinas, Edwin V. Bonilla
• Abstract summary: Deep graph neural networks (GNNs) suffer from over-smoothing when the number of layers increases. We propose DSGNNs enhanced with deep supervision where representations learned at all layers are used for training. We show that DSGNNs are resilient to over-smoothing and can outperform competitive benchmarks on node and graph property prediction problems.
• Score: 13.180922099929765
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning useful node and graph representations with graph neural networks (GNNs) is a challenging task. It is known that deep GNNs suffer from over-smoothing where, as the number of layers increases, node representations become nearly indistinguishable and model performance on the downstream task degrades significantly. To address this problem, we propose deeply-supervised GNNs (DSGNNs), i.e., GNNs enhanced with deep supervision where representations learned at all layers are used for training. We show empirically that DSGNNs are resilient to over-smoothing and can outperform competitive benchmarks on node and graph property prediction problems.

Related papers

• A Manifold Perspective on the Statistical Generalization of Graph Neural Networks [84.01980526069075]
  Graph Neural Networks (GNNs) combine information from adjacent nodes by successive applications of graph convolutions. We study the generalization gaps of GNNs on both node-level and graph-level tasks. We show that the generalization gaps decrease with the number of nodes in the training graphs.
  arXiv  Detail & Related papers  (2024-06-07T19:25:02Z)
• Information Flow in Graph Neural Networks: A Clinical Triage Use Case [49.86931948849343]
  Graph Neural Networks (GNNs) have gained popularity in healthcare and other domains due to their ability to process multi-modal and multi-relational graphs. We investigate how the flow of embedding information within GNNs affects the prediction of links in Knowledge Graphs (KGs) Our results demonstrate that incorporating domain knowledge into the GNN connectivity leads to better performance than using the same connectivity as the KG or allowing unconstrained embedding propagation.
  arXiv  Detail & Related papers  (2023-09-12T09:18:12Z)
• Towards Deep Attention in Graph Neural Networks: Problems and Remedies [15.36416000750147]
  Graph neural networks (GNNs) learn the representation of graph-structured data, and their expressiveness can be enhanced by inferring node relations for propagation. We investigate some problematic phenomena related to deep graph attention, including vulnerability to over-smoothed features and smooth cumulative attention. Motivated by our findings, we propose AEROGNN, a novel GNN architecture designed for deep graph attention.
  arXiv  Detail & Related papers  (2023-06-04T15:19:44Z)
• Distributed Graph Neural Network Training: A Survey [51.77035975191926]
  Graph neural networks (GNNs) are a type of deep learning models that are trained on graphs and have been successfully applied in various domains. Despite the effectiveness of GNNs, it is still challenging for GNNs to efficiently scale to large graphs. As a remedy, distributed computing becomes a promising solution of training large-scale GNNs.
  arXiv  Detail & Related papers  (2022-11-01T01:57:00Z)
• Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective [44.96635754139024]
  Oversmoothing has been identified as one of the key issues which limit the performance of deep GNNs. We propose a new perspective to look at the performance degradation of deep GNNs, i.e., feature overcorrelation. To reduce the feature correlation, we propose a general framework DeCorr which can encourage GNNs to encode less redundant information.
  arXiv  Detail & Related papers  (2022-06-15T18:13:52Z)
• CAP: Co-Adversarial Perturbation on Weights and Features for Improving Generalization of Graph Neural Networks [59.692017490560275]
  Adversarial training has been widely demonstrated to improve model's robustness against adversarial attacks. It remains unclear how the adversarial training could improve the generalization abilities of GNNs in the graph analytics problem. We construct the co-adversarial perturbation (CAP) optimization problem in terms of weights and features, and design the alternating adversarial perturbation algorithm to flatten the weight and feature loss landscapes alternately.
  arXiv  Detail & Related papers  (2021-10-28T02:28:13Z)
• Increase and Conquer: Training Graph Neural Networks on Growing Graphs [116.03137405192356]
  We consider the problem of learning a graphon neural network (WNN) by training GNNs on graphs sampled Bernoulli from the graphon. Inspired by these results, we propose an algorithm to learn GNNs on large-scale graphs that, starting from a moderate number of nodes, successively increases the size of the graph during training.
  arXiv  Detail & Related papers  (2021-06-07T15:05:59Z)
• AutoGraph: Automated Graph Neural Network [45.94642721490744]
  We propose a method to automate the deep Graph Neural Networks (GNNs) design. In our proposed method, we add a new type of skip connection to the GNNs search space to encourage feature reuse. We also allow our evolutionary algorithm to increase the layers of GNNs during the evolution to generate deeper networks.
  arXiv  Detail & Related papers  (2020-11-23T09:04:17Z)
• Attentive Graph Neural Networks for Few-Shot Learning [74.01069516079379]
  Graph Neural Networks (GNN) has demonstrated the superior performance in many challenging applications, including the few-shot learning tasks. Despite its powerful capacity to learn and generalize the model from few samples, GNN usually suffers from severe over-fitting and over-smoothing as the model becomes deep. We propose a novel Attentive GNN to tackle these challenges, by incorporating a triple-attention mechanism.
  arXiv  Detail & Related papers  (2020-07-14T07:43:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.