Generative and Contrastive Paradigms Are Complementary for Graph
Self-Supervised Learning
- URL: http://arxiv.org/abs/2310.15523v1
- Date: Tue, 24 Oct 2023 05:06:06 GMT
- Title: Generative and Contrastive Paradigms Are Complementary for Graph
Self-Supervised Learning
- Authors: Yuxiang Wang, Xiao Yan, Chuang Hu, Fangcheng Fu, Wentao Zhang, Hao
Wang, Shuo Shang, Jiawei Jiang
- Abstract summary: Masked autoencoder (MAE) learns to reconstruct masked graph edges or node features.
Contrastive Learning (CL) maximizes the similarity between augmented views of the same graph.
We propose graph contrastive masked autoencoder (GCMAE) framework to unify MAE and CL.
- Score: 56.45977379288308
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: For graph self-supervised learning (GSSL), masked autoencoder (MAE) follows
the generative paradigm and learns to reconstruct masked graph edges or node
features. Contrastive Learning (CL) maximizes the similarity between augmented
views of the same graph and is widely used for GSSL. However, MAE and CL are
considered separately in existing works for GSSL. We observe that the MAE and
CL paradigms are complementary and propose the graph contrastive masked
autoencoder (GCMAE) framework to unify them. Specifically, by focusing on local
edges or node features, MAE cannot capture global information of the graph and
is sensitive to particular edges and features. On the contrary, CL excels in
extracting global information because it considers the relation between graphs.
As such, we equip GCMAE with an MAE branch and a CL branch, and the two
branches share a common encoder, which allows the MAE branch to exploit the
global information extracted by the CL branch. To force GCMAE to capture global
graph structures, we train it to reconstruct the entire adjacency matrix
instead of only the masked edges as in existing works. Moreover, a
discrimination loss is proposed for feature reconstruction, which improves the
disparity between node embeddings rather than reducing the reconstruction error
to tackle the feature smoothing problem of MAE. We evaluate GCMAE on four
popular graph tasks (i.e., node classification, node clustering, link
prediction, and graph classification) and compare with 14 state-of-the-art
baselines. The results show that GCMAE consistently provides good accuracy
across these tasks, and the maximum accuracy improvement is up to 3.2% compared
with the best-performing baseline.
Related papers
- GRE^2-MDCL: Graph Representation Embedding Enhanced via Multidimensional Contrastive Learning [0.0]
Graph representation learning has emerged as a powerful tool for preserving graph topology when mapping nodes to vector representations.
Current graph neural network models face the challenge of requiring extensive labeled data.
We propose Graph Representation Embedding Enhanced via Multidimensional Contrastive Learning.
arXiv Detail & Related papers (2024-09-12T03:09:05Z) - Hi-GMAE: Hierarchical Graph Masked Autoencoders [90.30572554544385]
Hierarchical Graph Masked AutoEncoders (Hi-GMAE)
Hi-GMAE is a novel multi-scale GMAE framework designed to handle the hierarchical structures within graphs.
Our experiments on 15 graph datasets consistently demonstrate that Hi-GMAE outperforms 17 state-of-the-art self-supervised competitors.
arXiv Detail & Related papers (2024-05-17T09:08:37Z) - UGMAE: A Unified Framework for Graph Masked Autoencoders [67.75493040186859]
We propose UGMAE, a unified framework for graph masked autoencoders.
We first develop an adaptive feature mask generator to account for the unique significance of nodes.
We then design a ranking-based structure reconstruction objective joint with feature reconstruction to capture holistic graph information.
arXiv Detail & Related papers (2024-02-12T19:39:26Z) - Rethinking Graph Masked Autoencoders through Alignment and Uniformity [26.86368034133612]
Self-supervised learning on graphs can be bifurcated into contrastive and generative methods.
Recent advent of graph masked autoencoder (GraphMAE) rekindles momentum behind generative methods.
arXiv Detail & Related papers (2024-02-11T15:21:08Z) - GiGaMAE: Generalizable Graph Masked Autoencoder via Collaborative Latent
Space Reconstruction [76.35904458027694]
Masked autoencoder models lack good generalization ability on graph data.
We propose a novel graph masked autoencoder framework called GiGaMAE.
Our results will shed light on the design of foundation models on graph-structured data.
arXiv Detail & Related papers (2023-08-18T16:30:51Z) - What's Behind the Mask: Understanding Masked Graph Modeling for Graph
Autoencoders [32.42097625708298]
MaskGAE is a self-supervised learning framework for graph-structured data.
MGM is a principled pretext task - masking a portion of edges and attempting to reconstruct the missing part with partially visible, unmasked graph structure.
We establish close connections between GAEs and contrastive learning, showing that MGM significantly improves the self-supervised learning scheme of GAEs.
arXiv Detail & Related papers (2022-05-20T09:45:57Z) - MGAE: Masked Autoencoders for Self-Supervised Learning on Graphs [55.66953093401889]
Masked graph autoencoder (MGAE) framework to perform effective learning on graph structure data.
Taking insights from self-supervised learning, we randomly mask a large proportion of edges and try to reconstruct these missing edges during training.
arXiv Detail & Related papers (2022-01-07T16:48:07Z) - Graph Representation Learning via Contrasting Cluster Assignments [57.87743170674533]
We propose a novel unsupervised graph representation model by contrasting cluster assignments, called as GRCCA.
It is motivated to make good use of local and global information synthetically through combining clustering algorithms and contrastive learning.
GRCCA has strong competitiveness in most tasks.
arXiv Detail & Related papers (2021-12-15T07:28:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.