Tackling Oversmoothing of GNNs with Contrastive Learning
- URL: http://arxiv.org/abs/2110.13798v1
- Date: Tue, 26 Oct 2021 15:56:16 GMT
- Title: Tackling Oversmoothing of GNNs with Contrastive Learning
- Authors: Lecheng Zheng, Dongqi Fu, Jingrui He
- Abstract summary: Graph neural networks (GNNs) integrate the comprehensive relation of graph data and representation learning capability.
Oversmoothing makes the final representations of nodes indiscriminative, thus deteriorating the node classification and link prediction performance.
We propose the Topology-guided Graph Contrastive Layer, named TGCL, which is the first de-oversmoothing method maintaining all three mentioned metrics.
- Score: 35.88575306925201
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Graph neural networks (GNNs) integrate the comprehensive relation of graph
data and the representation learning capability of neural networks, which is
one of the most popular deep learning methods and achieves state-of-the-art
performance in many applications, such as natural language processing and
computer vision. In real-world scenarios, increasing the depth (i.e., the
number of layers) of GNNs is sometimes necessary to capture more latent
knowledge of the input data to mitigate the uncertainty caused by missing
values. However, involving more complex structures and more parameters will
decrease the performance of GNN models. One reason called oversmoothing is
recently introduced but the relevant research remains nascent. In general,
oversmoothing makes the final representations of nodes indiscriminative, thus
deteriorating the node classification and link prediction performance. In this
paper, we first survey the current de-oversmoothing methods and propose three
major metrics to evaluate a de-oversmoothing method, i.e., constant divergence
indicator, easy-to-determine divergence indicator, and model-agnostic strategy.
Then, we propose the Topology-guided Graph Contrastive Layer, named TGCL, which
is the first de-oversmoothing method maintaining all three mentioned metrics.
With the contrastive learning manner, we provide the theoretical analysis of
the effectiveness of the proposed TGCL. Last but not least, we design extensive
experiments to illustrate the empirical performance of TGCL comparing with
state-of-the-art baselines.
Related papers
- Breaking the Entanglement of Homophily and Heterophily in
Semi-supervised Node Classification [25.831508778029097]
We introduce AMUD, which quantifies the relationship between node profiles and topology from a statistical perspective.
We also propose ADPA as a new directed graph learning paradigm for AMUD.
arXiv Detail & Related papers (2023-12-07T07:54:11Z) - Label Deconvolution for Node Representation Learning on Large-scale
Attributed Graphs against Learning Bias [75.44877675117749]
We propose an efficient label regularization technique, namely Label Deconvolution (LD), to alleviate the learning bias by a novel and highly scalable approximation to the inverse mapping of GNNs.
Experiments demonstrate LD significantly outperforms state-of-the-art methods on Open Graph datasets Benchmark.
arXiv Detail & Related papers (2023-09-26T13:09:43Z) - Learning Strong Graph Neural Networks with Weak Information [64.64996100343602]
We develop a principled approach to the problem of graph learning with weak information (GLWI)
We propose D$2$PT, a dual-channel GNN framework that performs long-range information propagation on the input graph with incomplete structure, but also on a global graph that encodes global semantic similarities.
arXiv Detail & Related papers (2023-05-29T04:51:09Z) - DEGREE: Decomposition Based Explanation For Graph Neural Networks [55.38873296761104]
We propose DEGREE to provide a faithful explanation for GNN predictions.
By decomposing the information generation and aggregation mechanism of GNNs, DEGREE allows tracking the contributions of specific components of the input graph to the final prediction.
We also design a subgraph level interpretation algorithm to reveal complex interactions between graph nodes that are overlooked by previous methods.
arXiv Detail & Related papers (2023-05-22T10:29:52Z) - A Comprehensive Study on Large-Scale Graph Training: Benchmarking and
Rethinking [124.21408098724551]
Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs)
We present a new ensembling training manner, named EnGCN, to address the existing issues.
Our proposed method has achieved new state-of-the-art (SOTA) performance on large-scale datasets.
arXiv Detail & Related papers (2022-10-14T03:43:05Z) - Comprehensive Graph Gradual Pruning for Sparse Training in Graph Neural
Networks [52.566735716983956]
We propose a graph gradual pruning framework termed CGP to dynamically prune GNNs.
Unlike LTH-based methods, the proposed CGP approach requires no re-training, which significantly reduces the computation costs.
Our proposed strategy greatly improves both training and inference efficiency while matching or even exceeding the accuracy of existing methods.
arXiv Detail & Related papers (2022-07-18T14:23:31Z) - Graph Partner Neural Networks for Semi-Supervised Learning on Graphs [16.489177915147785]
Graph Convolutional Networks (GCNs) are powerful for processing graphstructured data and have achieved state-of-the-art performance in several tasks such as node classification, link prediction, and graph classification.
It is inevitable for deep GCNs to suffer from an over-smoothing issue that the representations of nodes will tend to be indistinguishable after repeated graph convolution operations.
We propose the Graph Partner Neural Network (GPNN) which incorporates a de- parameterized GCN and a parameter-sharing scheme.
arXiv Detail & Related papers (2021-10-18T10:56:56Z) - Fast Learning of Graph Neural Networks with Guaranteed Generalizability:
One-hidden-layer Case [93.37576644429578]
Graph neural networks (GNNs) have made great progress recently on learning from graph-structured data in practice.
We provide a theoretically-grounded generalizability analysis of GNNs with one hidden layer for both regression and binary classification problems.
arXiv Detail & Related papers (2020-06-25T00:45:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.