Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representations

• URL: http://arxiv.org/abs/2210.08345v1
• Date: Sat, 15 Oct 2022 17:36:04 GMT
• Title: Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representations
• Abstract summary: We propose an augmentation-free graph contrastive learning method, invariant-discriminative graph contrastive learning (iGCL) iGCL designs the invariant-discriminative loss (ID loss) to learn invariant and discriminative representations. Our theoretical analysis explains the effectiveness of ID loss from the perspectives of the redundancy reduction criterion, canonical correlation analysis, and information bottleneck principle.
• Score: 2.0621215354103635
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The pretasks are mainly built on mutual information estimation, which requires data augmentation to construct positive samples with similar semantics to learn invariant signals and negative samples with dissimilar semantics in order to empower representation discriminability. However, an appropriate data augmentation configuration depends heavily on lots of empirical trials such as choosing the compositions of data augmentation techniques and the corresponding hyperparameter settings. We propose an augmentation-free graph contrastive learning method, invariant-discriminative graph contrastive learning (iGCL), that does not intrinsically require negative samples. iGCL designs the invariant-discriminative loss (ID loss) to learn invariant and discriminative representations. On the one hand, ID loss learns invariant signals by directly minimizing the mean square error between the target samples and positive samples in the representation space. On the other hand, ID loss ensures that the representations are discriminative by an orthonormal constraint forcing the different dimensions of representations to be independent of each other. This prevents representations from collapsing to a point or subspace. Our theoretical analysis explains the effectiveness of ID loss from the perspectives of the redundancy reduction criterion, canonical correlation analysis, and information bottleneck principle. The experimental results demonstrate that iGCL outperforms all baselines on 5 node classification benchmark datasets. iGCL also shows superior performance for different label ratios and is capable of resisting graph attacks, which indicates that iGCL has excellent generalization and robustness.

Related papers

• Graph Structure Refinement with Energy-based Contrastive Learning [56.957793274727514]
  We introduce an unsupervised method based on a joint of generative training and discriminative training to learn graph structure and representation. We propose an Energy-based Contrastive Learning (ECL) guided Graph Structure Refinement (GSR) framework, denoted as ECL-GSR. ECL-GSR achieves faster training with fewer samples and memories against the leading baseline, highlighting its simplicity and efficiency in downstream tasks.
  arXiv  Detail & Related papers  (2024-12-20T04:05:09Z)
• Negative-Free Self-Supervised Gaussian Embedding of Graphs [29.26519601854811]
  Graph Contrastive Learning (GCL) has emerged as a promising graph self-supervised learning framework. We propose a negative-free objective to achieve uniformity, inspired by the fact that points distributed according to a normalized isotropic Gaussian are uniformly spread across the unit hypersphere. Our proposal achieves competitive performance with fewer parameters, shorter training times, and lower memory consumption compared to existing GCL methods.
  arXiv  Detail & Related papers  (2024-11-02T07:04:40Z)
• Topology Reorganized Graph Contrastive Learning with Mitigating Semantic Drift [28.83750578838018]
  Graph contrastive learning (GCL) is an effective paradigm for node representation learning in graphs. To increase the diversity of the contrastive view, we propose two simple and effective global topological augmentations to compensate current GCL.
  arXiv  Detail & Related papers  (2024-07-23T13:55:33Z)
• Prototypical Contrastive Learning through Alignment and Uniformity for Recommendation [6.790779112538357]
  We present underlinePrototypical contrastive learning through underlineAlignment and underlineUniformity for recommendation. Specifically, we first propose prototypes as a latent space to ensure consistency across different augmentations from the origin graph. The absence of explicit negatives means that directly optimizing the consistency loss between instance and prototype could easily result in dimensional collapse issues.
  arXiv  Detail & Related papers  (2024-02-03T08:19:26Z)
• Hodge-Aware Contrastive Learning [101.56637264703058]
  Simplicial complexes prove effective in modeling data with multiway dependencies. We develop a contrastive self-supervised learning approach for processing simplicial data.
  arXiv  Detail & Related papers  (2023-09-14T00:40:07Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• Single-Pass Contrastive Learning Can Work for Both Homophilic and Heterophilic Graph [60.28340453547902]
  Graph contrastive learning (GCL) techniques typically require two forward passes for a single instance to construct the contrastive loss. Existing GCL approaches fail to provide strong performance guarantees. We implement the Single-Pass Graph Contrastive Learning method (SP-GCL) Empirically, the features learned by the SP-GCL can match or outperform existing strong baselines with significantly less computational overhead.
  arXiv  Detail & Related papers  (2022-11-20T07:18:56Z)
• Rethinking Prototypical Contrastive Learning through Alignment, Uniformity and Correlation [24.794022951873156]
  We propose to learn Prototypical representation through Alignment, Uniformity and Correlation (PAUC) Specifically, the ordinary ProtoNCE loss is revised with: (1) an alignment loss that pulls embeddings from positive prototypes together; (2) a loss that distributes the prototypical level features uniformly; (3) a correlation loss that increases the diversity and discriminability between prototypical level features.
  arXiv  Detail & Related papers  (2022-10-18T22:33:12Z)
• Hierarchical Semi-Supervised Contrastive Learning for Contamination-Resistant Anomaly Detection [81.07346419422605]
  Anomaly detection aims at identifying deviant samples from the normal data distribution. Contrastive learning has provided a successful way to sample representation that enables effective discrimination on anomalies. We propose a novel hierarchical semi-supervised contrastive learning framework, for contamination-resistant anomaly detection.
  arXiv  Detail & Related papers  (2022-07-24T18:49:26Z)
• Prototypical Graph Contrastive Learning [141.30842113683775]
  We propose a Prototypical Graph Contrastive Learning (PGCL) approach to mitigate the critical sampling bias issue. Specifically, PGCL models the underlying semantic structure of the graph data via clustering semantically similar graphs into the same group, and simultaneously encourages the clustering consistency for different augmentations of the same graph. For a query, PGCL further reweights its negative samples based on the distance between their prototypes (cluster centroids) and the query prototype.
  arXiv  Detail & Related papers  (2021-06-17T16:45:31Z)
• Learning Invariant Representations using Inverse Contrastive Loss [34.93395633215398]
  We introduce a class of losses for learning representations that are invariant to some extraneous variable of interest. We show that if the extraneous variable is binary, then optimizing ICL is equivalent to optimizing a regularized MMD divergence.
  arXiv  Detail & Related papers  (2021-02-16T18:29:28Z)
• When Relation Networks meet GANs: Relation GANs with Triplet Loss [110.7572918636599]
  Training stability is still a lingering concern of generative adversarial networks (GANs) In this paper, we explore a relation network architecture for the discriminator and design a triplet loss which performs better generalization and stability. Experiments on benchmark datasets show that the proposed relation discriminator and new loss can provide significant improvement on variable vision tasks.
  arXiv  Detail & Related papers  (2020-02-24T11:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.