Negative-Free Self-Supervised Gaussian Embedding of Graphs

• URL: http://arxiv.org/abs/2411.01157v1
• Date: Sat, 02 Nov 2024 07:04:40 GMT
• Title: Negative-Free Self-Supervised Gaussian Embedding of Graphs
• Authors: Yunhui Liu, Tieke He, Tao Zheng, Jianhua Zhao,
• Abstract summary: Graph Contrastive Learning (GCL) has emerged as a promising graph self-supervised learning framework. We propose a negative-free objective to achieve uniformity, inspired by the fact that points distributed according to a normalized isotropic Gaussian are uniformly spread across the unit hypersphere. Our proposal achieves competitive performance with fewer parameters, shorter training times, and lower memory consumption compared to existing GCL methods.
• Score: 29.26519601854811
• License:
• Abstract: Graph Contrastive Learning (GCL) has recently emerged as a promising graph self-supervised learning framework for learning discriminative node representations without labels. The widely adopted objective function of GCL benefits from two key properties: \emph{alignment} and \emph{uniformity}, which align representations of positive node pairs while uniformly distributing all representations on the hypersphere. The uniformity property plays a critical role in preventing representation collapse and is achieved by pushing apart augmented views of different nodes (negative pairs). As such, existing GCL methods inherently rely on increasing the quantity and quality of negative samples, resulting in heavy computational demands, memory overhead, and potential class collision issues. In this study, we propose a negative-free objective to achieve uniformity, inspired by the fact that points distributed according to a normalized isotropic Gaussian are uniformly spread across the unit hypersphere. Therefore, we can minimize the distance between the distribution of learned representations and the isotropic Gaussian distribution to promote the uniformity of node representations. Our method also distinguishes itself from other approaches by eliminating the need for a parameterized mutual information estimator, an additional projector, asymmetric structures, and, crucially, negative samples. Extensive experiments over seven graph benchmarks demonstrate that our proposal achieves competitive performance with fewer parameters, shorter training times, and lower memory consumption compared to existing GCL methods.

Related papers

• Bootstrap Latents of Nodes and Neighbors for Graph Self-Supervised Learning [27.278097015083343]
  Contrastive learning requires negative samples to prevent model collapse and learn discriminative representations. We introduce a cross-attention module to predict the supportiveness score of a neighbor with respect to the anchor node. Our method mitigates class collision from negative and noisy positive samples, concurrently enhancing intra-class compactness.
  arXiv  Detail & Related papers  (2024-08-09T14:17:52Z)
• Topology Reorganized Graph Contrastive Learning with Mitigating Semantic Drift [28.83750578838018]
  Graph contrastive learning (GCL) is an effective paradigm for node representation learning in graphs. To increase the diversity of the contrastive view, we propose two simple and effective global topological augmentations to compensate current GCL.
  arXiv  Detail & Related papers  (2024-07-23T13:55:33Z)
• Smoothed Graph Contrastive Learning via Seamless Proximity Integration [35.73306919276754]
  Graph contrastive learning (GCL) aligns node representations by classifying node pairs into positives and negatives. We present a Smoothed Graph Contrastive Learning model (SGCL) that injects proximity information associated with positive/negative pairs in the contrastive loss. The proposed SGCL adjusts the penalties associated with node pairs in the contrastive loss by incorporating three distinct smoothing techniques.
  arXiv  Detail & Related papers  (2024-02-23T11:32:46Z)
• STERLING: Synergistic Representation Learning on Bipartite Graphs [78.86064828220613]
  A fundamental challenge of bipartite graph representation learning is how to extract node embeddings. Most recent bipartite graph SSL methods are based on contrastive learning which learns embeddings by discriminating positive and negative node pairs. We introduce a novel synergistic representation learning model (STERLING) to learn node embeddings without negative node pairs.
  arXiv  Detail & Related papers  (2023-01-25T03:21:42Z)
• Localized Contrastive Learning on Graphs [110.54606263711385]
  We introduce a simple yet effective contrastive model named Localized Graph Contrastive Learning (Local-GCL) In spite of its simplicity, Local-GCL achieves quite competitive performance in self-supervised node representation learning tasks on graphs with various scales and properties.
  arXiv  Detail & Related papers  (2022-12-08T23:36:00Z)
• Single-Pass Contrastive Learning Can Work for Both Homophilic and Heterophilic Graph [60.28340453547902]
  Graph contrastive learning (GCL) techniques typically require two forward passes for a single instance to construct the contrastive loss. Existing GCL approaches fail to provide strong performance guarantees. We implement the Single-Pass Graph Contrastive Learning method (SP-GCL) Empirically, the features learned by the SP-GCL can match or outperform existing strong baselines with significantly less computational overhead.
  arXiv  Detail & Related papers  (2022-11-20T07:18:56Z)
• Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representations [2.0621215354103635]
  We propose an augmentation-free graph contrastive learning method, invariant-discriminative graph contrastive learning (iGCL) iGCL designs the invariant-discriminative loss (ID loss) to learn invariant and discriminative representations. Our theoretical analysis explains the effectiveness of ID loss from the perspectives of the redundancy reduction criterion, canonical correlation analysis, and information bottleneck principle.
  arXiv  Detail & Related papers  (2022-10-15T17:36:04Z)
• Geometric Graph Representation Learning via Maximizing Rate Reduction [73.6044873825311]
  Learning node representations benefits various downstream tasks in graph analysis such as community detection and node classification. We propose Geometric Graph Representation Learning (G2R) to learn node representations in an unsupervised manner. G2R maps nodes in distinct groups into different subspaces, while each subspace is compact and different subspaces are dispersed.
  arXiv  Detail & Related papers  (2022-02-13T07:46:24Z)
• Exploring Non-Contrastive Representation Learning for Deep Clustering [23.546602131801205]
  Non-contrastive representation learning for deep clustering, termed NCC, is based on BYOL, a representative method without negative examples. NCC forms an embedding space where all clusters are well-separated and within-cluster examples are compact. Experimental results on several clustering benchmark datasets including ImageNet-1K demonstrate that NCC outperforms the state-of-the-art methods by a significant margin.
  arXiv  Detail & Related papers  (2021-11-23T12:21:53Z)
• Prototypical Graph Contrastive Learning [141.30842113683775]
  We propose a Prototypical Graph Contrastive Learning (PGCL) approach to mitigate the critical sampling bias issue. Specifically, PGCL models the underlying semantic structure of the graph data via clustering semantically similar graphs into the same group, and simultaneously encourages the clustering consistency for different augmentations of the same graph. For a query, PGCL further reweights its negative samples based on the distance between their prototypes (cluster centroids) and the query prototype.
  arXiv  Detail & Related papers  (2021-06-17T16:45:31Z)
• Block-Approximated Exponential Random Graphs [77.4792558024487]
  An important challenge in the field of exponential random graphs (ERGs) is the fitting of non-trivial ERGs on large graphs. We propose an approximative framework to such non-trivial ERGs that result in dyadic independence (i.e., edge independent) distributions. Our methods are scalable to sparse graphs consisting of millions of nodes.
  arXiv  Detail & Related papers  (2020-02-14T11:42:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.