Understanding and Resolving Performance Degradation in Graph Convolutional Networks

• URL: http://arxiv.org/abs/2006.07107v3
• Date: Mon, 13 Sep 2021 15:11:26 GMT
• Title: Understanding and Resolving Performance Degradation in Graph Convolutional Networks
• Authors: Kuangqi Zhou, Yanfei Dong, Kaixin Wang, Wee Sun Lee, Bryan Hooi, Huan Xu, Jiashi Feng
• Abstract summary: Graph Convolutional Network (GCN) stacks several layers and in each layer performs a PROPagation operation (PROP) and a TRANsformation operation (TRAN) for learning node representations over graph-structured data. GCNs tend to suffer performance drop when the model gets deep. We study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works.
• Score: 105.14867349802898
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A Graph Convolutional Network (GCN) stacks several layers and in each layer performs a PROPagation operation (PROP) and a TRANsformation operation (TRAN) for learning node representations over graph-structured data. Though powerful, GCNs tend to suffer performance drop when the model gets deep. Previous works focus on PROPs to study and mitigate this issue, but the role of TRANs is barely investigated. In this work, we study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works. We find that TRANs contribute significantly, or even more than PROPs, to declining performance, and moreover that they tend to amplify node-wise feature variance in GCNs, causing variance inflammation that we identify as a key factor for causing performance drop. Motivated by such observations, we propose a variance-controlling technique termed Node Normalization (NodeNorm), which scales each node's features using its own standard deviation. Experimental results validate the effectiveness of NodeNorm on addressing performance degradation of GCNs. Specifically, it enables deep GCNs to outperform shallow ones in cases where deep models are needed, and to achieve comparable results with shallow ones on 6 benchmark datasets. NodeNorm is a generic plug-in and can well generalize to other GNN architectures. Code is publicly available at https://github.com/miafei/NodeNorm.

Related papers

• DRGCN: Dynamic Evolving Initial Residual for Deep Graph Convolutional Networks [19.483662490506646]
  We propose a novel model called Dynamic evolving initial Residual Graph Convolutional Network (DRGCN) Our experimental results show that our model effectively relieves the problem of over-smoothing in deep GCNs. Our model reaches new SOTA results on the large-scale ogbn-arxiv dataset of Open Graph Benchmark (OGB)
  arXiv  Detail & Related papers  (2023-02-10T06:57:12Z)
• ResNorm: Tackling Long-tailed Degree Distribution Issue in Graph Neural Networks via Normalization [80.90206641975375]
  This paper focuses on improving the performance of GNNs via normalization. By studying the long-tailed distribution of node degrees in the graph, we propose a novel normalization method for GNNs. The $scale$ operation of ResNorm reshapes the node-wise standard deviation (NStd) distribution so as to improve the accuracy of tail nodes.
  arXiv  Detail & Related papers  (2022-06-16T13:49:09Z)
• SkipNode: On Alleviating Performance Degradation for Deep Graph Convolutional Networks [84.30721808557871]
  We conduct theoretical and experimental analysis to explore the fundamental causes of performance degradation in deep GCNs. We propose a simple yet effective plug-and-play module, Skipnode, to overcome the performance degradation of deep GCNs.
  arXiv  Detail & Related papers  (2021-12-22T02:18:31Z)
• Node Feature Kernels Increase Graph Convolutional Network Robustness [19.076912727990326]
  The robustness of Graph Convolutional Networks (GCNs) to perturbations of their input is becoming a topic of increasing importance. In this paper, a random matrix theory analysis is possible. It is observed that enhancing the message passing step in GCNs by adding the node feature kernel to the adjacency matrix of the graph structure solves this problem.
  arXiv  Detail & Related papers  (2021-09-04T04:20:45Z)
• Tackling Over-Smoothing for General Graph Convolutional Networks [88.71154017107257]
  We study how general GCNs act with the increase in depth, including generic GCN, GCN with bias, ResGCN, and APPNP. We propose DropEdge to alleviate over-smoothing by randomly removing a certain number of edges at each training epoch.
  arXiv  Detail & Related papers  (2020-08-22T16:14:01Z)
• DeeperGCN: All You Need to Train Deeper GCNs [66.64739331859226]
  Graph Convolutional Networks (GCNs) have been drawing significant attention with the power of representation learning on graphs. Unlike Convolutional Neural Networks (CNNs), which are able to take advantage of stacking very deep layers, GCNs suffer from vanishing gradient, over-smoothing and over-fitting issues when going deeper. This paper proposes DeeperGCN that is capable of successfully and reliably training very deep GCNs.
  arXiv  Detail & Related papers  (2020-06-13T23:00:22Z)
• Towards Deeper Graph Neural Networks with Differentiable Group Normalization [61.20639338417576]
  Graph neural networks (GNNs) learn the representation of a node by aggregating its neighbors. Over-smoothing is one of the key issues which limit the performance of GNNs as the number of layers increases. We introduce two over-smoothing metrics and a novel technique, i.e., differentiable group normalization (DGN)
  arXiv  Detail & Related papers  (2020-06-12T07:18:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.