SkipNode: On Alleviating Performance Degradation for Deep Graph Convolutional Networks

• URL: http://arxiv.org/abs/2112.11628v4
• Date: Tue, 23 Jan 2024 03:03:54 GMT
• Title: SkipNode: On Alleviating Performance Degradation for Deep Graph Convolutional Networks
• Authors: Weigang Lu, Yibing Zhan, Binbin Lin, Ziyu Guan, Liu Liu, Baosheng Yu, Wei Zhao, Yaming Yang, and Dacheng Tao
• Abstract summary: We conduct theoretical and experimental analysis to explore the fundamental causes of performance degradation in deep GCNs. We propose a simple yet effective plug-and-play module, Skipnode, to overcome the performance degradation of deep GCNs.
• Score: 84.30721808557871
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graph Convolutional Networks (GCNs) suffer from performance degradation when models go deeper. However, earlier works only attributed the performance degeneration to over-smoothing. In this paper, we conduct theoretical and experimental analysis to explore the fundamental causes of performance degradation in deep GCNs: over-smoothing and gradient vanishing have a mutually reinforcing effect that causes the performance to deteriorate more quickly in deep GCNs. On the other hand, existing anti-over-smoothing methods all perform full convolutions up to the model depth. They could not well resist the exponential convergence of over-smoothing due to model depth increasing. In this work, we propose a simple yet effective plug-and-play module, Skipnode, to overcome the performance degradation of deep GCNs. It samples graph nodes in each convolutional layer to skip the convolution operation. In this way, both over-smoothing and gradient vanishing can be effectively suppressed since (1) not all nodes'features propagate through full layers and, (2) the gradient can be directly passed back through ``skipped'' nodes. We provide both theoretical analysis and empirical evaluation to demonstrate the efficacy of Skipnode and its superiority over SOTA baselines.

Related papers

• Taming Gradient Oversmoothing and Expansion in Graph Neural Networks [3.0764244780817283]
  Oversmoothing has been claimed as a primary bottleneck for graph neural networks (GNNs) We show the presence of $textitgradient oversmoothing$ preventing optimization during training. We provide a simple yet effective normalization method to prevent the gradient expansion.
  arXiv  Detail & Related papers  (2024-10-07T08:22:20Z)
• Node Duplication Improves Cold-start Link Prediction [52.917775253887264]
  Graph Neural Networks (GNNs) are prominent in graph machine learning. Recent studies show that GNNs struggle to produce good results on low-degree nodes. We propose a simple yet surprisingly effective augmentation technique called NodeDup.
  arXiv  Detail & Related papers  (2024-02-15T05:07:39Z)
• A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networks [33.35609077417775]
  We characterize the mechanism behind the phenomenon via a non-asymptotic analysis. We show that oversmoothing happens once the mixing effect starts to dominate the denoising effect. Our results suggest that while PPR mitigates oversmoothing at deeper layers, PPR-based architectures still achieve their best performance at a shallow depth.
  arXiv  Detail & Related papers  (2022-12-21T00:33:59Z)
• Deep Manifold Learning with Graph Mining [80.84145791017968]
  We propose a novel graph deep model with a non-gradient decision layer for graph mining. The proposed model has achieved state-of-the-art performance compared to the current models.
  arXiv  Detail & Related papers  (2022-07-18T04:34:08Z)
• Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective [10.178145000390671]
  Graph convolutional networks (GCNs) have achieved promising performance on various graph-based tasks. However they suffer from over-smoothing when stacking more layers. We present a quantitative study on this observation and develop novel insights towards the deeper GCN.
  arXiv  Detail & Related papers  (2020-09-24T03:36:43Z)
• Tackling Over-Smoothing for General Graph Convolutional Networks [88.71154017107257]
  We study how general GCNs act with the increase in depth, including generic GCN, GCN with bias, ResGCN, and APPNP. We propose DropEdge to alleviate over-smoothing by randomly removing a certain number of edges at each training epoch.
  arXiv  Detail & Related papers  (2020-08-22T16:14:01Z)
• Simple and Deep Graph Convolutional Networks [63.76221532439285]
  Graph convolutional networks (GCNs) are a powerful deep learning approach for graph-structured data. Despite their success, most of the current GCN models are shallow, due to the em over-smoothing problem. We propose the GCNII, an extension of the vanilla GCN model with two simple yet effective techniques.
  arXiv  Detail & Related papers  (2020-07-04T16:18:06Z)
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks [105.14867349802898]
  Graph Convolutional Network (GCN) stacks several layers and in each layer performs a PROPagation operation (PROP) and a TRANsformation operation (TRAN) for learning node representations over graph-structured data. GCNs tend to suffer performance drop when the model gets deep. We study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works.
  arXiv  Detail & Related papers  (2020-06-12T12:12:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.