Rethinking Spectral Augmentation for Contrast-based Graph Self-Supervised Learning

• URL: http://arxiv.org/abs/2405.19600v2
• Date: Wed, 04 Dec 2024 04:41:49 GMT
• Title: Rethinking Spectral Augmentation for Contrast-based Graph Self-Supervised Learning
• Authors: Xiangru Jian, Xinjian Zhao, Wei Pang, Chaolong Ying, Yimu Wang, Yaoyao Xu, Tianshu Yu,
• Abstract summary: Methods grounded in seemingly conflicting assumptions regarding the spectral domain demonstrate notable enhancements in learning performance.<n>This suggests that the computational overhead of sophisticated spectral augmentations may not justify their practical benefits.<n>The proposed insights represent a significant leap forward in the field, potentially refining the understanding and implementation of graph self-supervised learning.
• Score: 10.803503272887173
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The recent surge in contrast-based graph self-supervised learning has prominently featured an intensified exploration of spectral cues. Spectral augmentation, which involves modifying a graph's spectral properties such as eigenvalues or eigenvectors, is widely believed to enhance model performance. However, an intriguing paradox emerges, as methods grounded in seemingly conflicting assumptions regarding the spectral domain demonstrate notable enhancements in learning performance. Through extensive empirical studies, we find that simple edge perturbations - random edge dropping for node-level and random edge adding for graph-level self-supervised learning - consistently yield comparable or superior performance while being significantly more computationally efficient. This suggests that the computational overhead of sophisticated spectral augmentations may not justify their practical benefits. Our theoretical analysis of the InfoNCE loss bounds for shallow GNNs further supports this observation. The proposed insights represent a significant leap forward in the field, potentially refining the understanding and implementation of graph self-supervised learning.

Related papers

• AS-GCL: Asymmetric Spectral Augmentation on Graph Contrastive Learning [25.07818336162072]
  Graph Contrastive Learning (GCL) has emerged as the foremost approach for self-supervised learning on graph-structured data. We propose a novel paradigm called AS-GCL that incorporates asymmetric spectral augmentation for graph contrastive learning. Our method introduces significant enhancements to each of these components.
  arXiv  Detail & Related papers  (2025-02-19T08:22:57Z)
• Consistency of augmentation graph and network approximability in contrastive learning [3.053989095162017]
  We analyze the pointwise and spectral consistency of the augmentation graph Laplacian. We show that Laplacian converges to a weighted Laplace-Beltrami operator on the natural data manifold. These consistency results ensure that the graph Laplacian spectrum effectively captures the manifold geometry.
  arXiv  Detail & Related papers  (2025-02-06T18:55:51Z)
• Disentangled Generative Graph Representation Learning [51.59824683232925]
  This paper introduces DiGGR (Disentangled Generative Graph Representation Learning), a self-supervised learning framework. It aims to learn latent disentangled factors and utilize them to guide graph mask modeling. Experiments on 11 public datasets for two different graph learning tasks demonstrate that DiGGR consistently outperforms many previous self-supervised methods.
  arXiv  Detail & Related papers  (2024-08-24T05:13:02Z)
• Through the Dual-Prism: A Spectral Perspective on Graph Data Augmentation for Graph Classification [67.35058947477631]
  We introduce Dual-Prism (DP) augmentation methods, including DP-Noise and DP-Mask, which retain essential graph properties while diversifying augmented graphs. Extensive experiments validate the efficiency of our approach, providing a new and promising direction for graph data augmentation.
  arXiv  Detail & Related papers  (2024-01-18T12:58:53Z)
• Graph-level Protein Representation Learning by Structure Knowledge Refinement [50.775264276189695]
  This paper focuses on learning representation on the whole graph level in an unsupervised manner. We propose a novel framework called Structure Knowledge Refinement (SKR) which uses data structure to determine the probability of whether a pair is positive or negative.
  arXiv  Detail & Related papers  (2024-01-05T09:05:33Z)
• Understanding Community Bias Amplification in Graph Representation Learning [22.522798932536038]
  We study a phenomenon of community bias amplification in graph representation learning. We propose a novel graph contrastive learning model called Random Graph Coarsening Contrastive Learning.
  arXiv  Detail & Related papers  (2023-12-08T07:43:05Z)
• Spectral Augmentations for Graph Contrastive Learning [50.149996923976836]
  Contrastive learning has emerged as a premier method for learning representations with or without supervision. Recent studies have shown its utility in graph representation learning for pre-training. We propose a set of well-motivated graph transformation operations to provide a bank of candidates when constructing augmentations for a graph contrastive objective.
  arXiv  Detail & Related papers  (2023-02-06T16:26:29Z)
• Understanding Self-Predictive Learning for Reinforcement Learning [61.62067048348786]
  We study the learning dynamics of self-predictive learning for reinforcement learning. We propose a novel self-predictive algorithm that learns two representations simultaneously.
  arXiv  Detail & Related papers  (2022-12-06T20:43:37Z)
• Spectral Feature Augmentation for Graph Contrastive Learning and Beyond [64.78221638149276]
  We present a novel spectral feature argumentation for contrastive learning on graphs (and images) For each data view, we estimate a low-rank approximation per feature map and subtract that approximation from the map to obtain its complement. This is achieved by the proposed herein incomplete power iteration, a non-standard power regime which enjoys two valuable byproducts (under mere one or two iterations) Experiments on graph/image datasets show that our spectral feature augmentation outperforms baselines.
  arXiv  Detail & Related papers  (2022-12-02T08:48:11Z)
• Spectral Augmentation for Self-Supervised Learning on Graphs [43.19199994575821]
  Graph contrastive learning (GCL) aims to learn representations via instance discrimination. It relies on graph augmentation to reflect invariant patterns that are robust to small perturbations. Recent studies mainly perform topology augmentations in a uniformly random manner in the spatial domain. We develop spectral augmentation which guides topology augmentations by maximizing the spectral change.
  arXiv  Detail & Related papers  (2022-10-02T22:20:07Z)
• Robust Causal Graph Representation Learning against Confounding Effects [21.380907101361643]
  We propose Robust Causal Graph Representation Learning (RCGRL) to learn robust graph representations against confounding effects. RCGRL introduces an active approach to generate instrumental variables under unconditional moment restrictions, which empowers the graph representation learning model to eliminate confounders.
  arXiv  Detail & Related papers  (2022-08-18T01:31:25Z)
• Learning node embeddings via summary graphs: a brief theoretical analysis [55.25628709267215]
  Graph representation learning plays an important role in many graph mining applications, but learning embeddings of large-scale graphs remains a problem. Recent works try to improve scalability via graph summarization -- i.e., they learn embeddings on a smaller summary graph, and then restore the node embeddings of the original graph. We give an in-depth theoretical analysis of three specific embedding learning methods based on introduced kernel matrix.
  arXiv  Detail & Related papers  (2022-07-04T04:09:50Z)
• Latent Augmentation For Better Graph Self-Supervised Learning [20.082614919182692]
  We argue that predictive models weaponed with latent augmentations and powerful decoder could achieve comparable or even better representation power than contrastive models. A novel graph decoder named Wiener Graph Deconvolutional Network is correspondingly designed to perform information reconstruction from augmented latent representations.
  arXiv  Detail & Related papers  (2022-06-26T17:41:59Z)
• Bayesian Graph Contrastive Learning [55.36652660268726]
  We propose a novel perspective of graph contrastive learning methods showing random augmentations leads to encoders. Our proposed method represents each node by a distribution in the latent space in contrast to existing techniques which embed each node to a deterministic vector. We show a considerable improvement in performance compared to existing state-of-the-art methods on several benchmark datasets.
  arXiv  Detail & Related papers  (2021-12-15T01:45:32Z)
• Fairness-Aware Node Representation Learning [9.850791193881651]
  This study addresses fairness issues in graph contrastive learning with fairness-aware graph augmentation designs. Different fairness notions on graphs are introduced, which serve as guidelines for the proposed graph augmentations. Experimental results on real social networks are presented to demonstrate that the proposed augmentations can enhance fairness in terms of statistical parity and equal opportunity.
  arXiv  Detail & Related papers  (2021-06-09T21:12:14Z)
• Iterative Graph Self-Distillation [161.04351580382078]
  We propose a novel unsupervised graph learning paradigm called Iterative Graph Self-Distillation (IGSD) IGSD iteratively performs the teacher-student distillation with graph augmentations. We show that we achieve significant and consistent performance gain on various graph datasets in both unsupervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-10-23T18:37:06Z)
• Towards Deeper Graph Neural Networks [63.46470695525957]
  Graph convolutions perform neighborhood aggregation and represent one of the most important graph operations. Several recent studies attribute this performance deterioration to the over-smoothing issue. We propose Deep Adaptive Graph Neural Network (DAGNN) to adaptively incorporate information from large receptive fields.
  arXiv  Detail & Related papers  (2020-07-18T01:11:14Z)
• Spectral Graph Attention Network with Fast Eigen-approximation [103.93113062682633]
  Spectral Graph Attention Network (SpGAT) learns representations for different frequency components regarding weighted filters and graph wavelets bases. Fast approximation variant SpGAT-Cheby is proposed to reduce the computational cost brought by the eigen-decomposition. We thoroughly evaluate the performance of SpGAT and SpGAT-Cheby in semi-supervised node classification tasks.
  arXiv  Detail & Related papers  (2020-03-16T21:49:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.