Provable Training for Graph Contrastive Learning

• URL: http://arxiv.org/abs/2309.13944v4
• Date: Fri, 24 May 2024 07:05:30 GMT
• Title: Provable Training for Graph Contrastive Learning
• Authors: Yue Yu, Xiao Wang, Mengmei Zhang, Nian Liu, Chuan Shi,
• Abstract summary: Graph Contrastive Learning (GCL) has emerged as a popular training approach for learning node embeddings from augmented graphs without labels. We show that the training of GCL is indeed imbalanced across all nodes. We propose the metric "node compactness", which is the lower bound of how a node follows the GCL principle.
• Score: 58.8128675529977
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graph Contrastive Learning (GCL) has emerged as a popular training approach for learning node embeddings from augmented graphs without labels. Despite the key principle that maximizing the similarity between positive node pairs while minimizing it between negative node pairs is well established, some fundamental problems are still unclear. Considering the complex graph structure, are some nodes consistently well-trained and following this principle even with different graph augmentations? Or are there some nodes more likely to be untrained across graph augmentations and violate the principle? How to distinguish these nodes and further guide the training of GCL? To answer these questions, we first present experimental evidence showing that the training of GCL is indeed imbalanced across all nodes. To address this problem, we propose the metric "node compactness", which is the lower bound of how a node follows the GCL principle related to the range of augmentations. We further derive the form of node compactness theoretically through bound propagation, which can be integrated into binary cross-entropy as a regularization. To this end, we propose the PrOvable Training (POT) for GCL, which regularizes the training of GCL to encode node embeddings that follows the GCL principle better. Through extensive experiments on various benchmarks, POT consistently improves the existing GCL approaches, serving as a friendly plugin.

Related papers

• Scalable Graph Compressed Convolutions [68.85227170390864]
  We propose a differentiable method that applies permutations to calibrate input graphs for Euclidean convolution. Based on the graph calibration, we propose the Compressed Convolution Network (CoCN) for hierarchical graph representation learning.
  arXiv  Detail & Related papers  (2024-07-26T03:14:13Z)
• Smoothed Graph Contrastive Learning via Seamless Proximity Integration [30.247207861739245]
  Graph contrastive learning (GCL) aligns node representations by classifying node pairs into positives and negatives. We present a Smoothed Graph Contrastive Learning model (SGCL) that injects proximity information associated with positive/negative pairs in the contrastive loss. The proposed SGCL adjusts the penalties associated with node pairs in contrastive loss by incorporating three distinct smoothing techniques.
  arXiv  Detail & Related papers  (2024-02-23T11:32:46Z)
• HomoGCL: Rethinking Homophily in Graph Contrastive Learning [64.85392028383164]
  HomoGCL is a model-agnostic framework to expand the positive set using neighbor nodes with neighbor-specific significances. We show that HomoGCL yields multiple state-of-the-art results across six public datasets.
  arXiv  Detail & Related papers  (2023-06-16T04:06:52Z)
• NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification [70.51126383984555]
  We introduce a novel all-pair message passing scheme for efficiently propagating node signals between arbitrary nodes. The efficient computation is enabled by a kernerlized Gumbel-Softmax operator. Experiments demonstrate the promising efficacy of the method in various tasks including node classification on graphs.
  arXiv  Detail & Related papers  (2023-06-14T09:21:15Z)
• Structural Imbalance Aware Graph Augmentation Learning [2.793446335600599]
  Graphs are often structurally imbalanced, that is, only a few hub nodes have a denser local structure and higher influence. This paper proposes a selective graph augmentation method (SAug) to solve this problem. Extensive experiments demonstrate that SAug can significantly improve the backbone GNNs and achieve superior performance to its competitors.
  arXiv  Detail & Related papers  (2023-03-24T02:13:32Z)
• Uncovering the Structural Fairness in Graph Contrastive Learning [87.65091052291544]
  Graph contrastive learning (GCL) has emerged as a promising self-supervised approach for learning node representations. We show that representations obtained by GCL methods are already fairer to degree bias than those learned by GCN. We devise a novel graph augmentation method, called GRAph contrastive learning for DEgree bias (GRADE), which applies different strategies to low- and high-degree nodes.
  arXiv  Detail & Related papers  (2022-10-06T15:58:25Z)
• ImGCL: Revisiting Graph Contrastive Learning on Imbalanced Node Classification [26.0350727426613]
  Graph contrastive learning (GCL) has attracted a surge of attention due to its superior performance for learning node/graph representations without labels. In practice, the underlying class distribution of unlabeled nodes for the given graph is usually imbalanced. We propose a principled GCL framework on Imbalanced node classification (ImGCL), which automatically and adaptively balances the representations learned from GCL without labels.
  arXiv  Detail & Related papers  (2022-05-23T14:23:36Z)
• Graph Inference Learning for Semi-supervised Classification [50.55765399527556]
  We propose a Graph Inference Learning framework to boost the performance of semi-supervised node classification. For learning the inference process, we introduce meta-optimization on structure relations from training nodes to validation nodes. Comprehensive evaluations on four benchmark datasets demonstrate the superiority of our proposed GIL when compared against state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-01-17T02:52:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.