A Message Passing Perspective on Learning Dynamics of Contrastive Learning

• URL: http://arxiv.org/abs/2303.04435v1
• Date: Wed, 8 Mar 2023 08:27:31 GMT
• Title: A Message Passing Perspective on Learning Dynamics of Contrastive Learning
• Authors: Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang, Zhouchen Lin, Yisen Wang
• Abstract summary: We show that if we cast a contrastive objective equivalently into the feature space, then its learning dynamics admits an interpretable form. This perspective also establishes an intriguing connection between contrastive learning and Message Passing Graph Neural Networks (MP-GNNs)
• Score: 60.217972614379065
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, contrastive learning achieves impressive results on self-supervised visual representation learning, but there still lacks a rigorous understanding of its learning dynamics. In this paper, we show that if we cast a contrastive objective equivalently into the feature space, then its learning dynamics admits an interpretable form. Specifically, we show that its gradient descent corresponds to a specific message passing scheme on the corresponding augmentation graph. Based on this perspective, we theoretically characterize how contrastive learning gradually learns discriminative features with the alignment update and the uniformity update. Meanwhile, this perspective also establishes an intriguing connection between contrastive learning and Message Passing Graph Neural Networks (MP-GNNs). This connection not only provides a unified understanding of many techniques independently developed in each community, but also enables us to borrow techniques from MP-GNNs to design new contrastive learning variants, such as graph attention, graph rewiring, jumpy knowledge techniques, etc. We believe that our message passing perspective not only provides a new theoretical understanding of contrastive learning dynamics, but also bridges the two seemingly independent areas together, which could inspire more interleaving studies to benefit from each other. The code is available at https://github.com/PKU-ML/Message-Passing-Contrastive-Learning.

Related papers

• GPS: Graph Contrastive Learning via Multi-scale Augmented Views from Adversarial Pooling [23.450755275125577]
  Self-supervised graph representation learning has recently shown considerable promise in a range of fields, including bioinformatics and social networks. We present a novel approach named Graph Pooling ContraSt (GPS) to address these issues. Motivated by the fact that graph pooling can adaptively coarsen the graph with the removal of redundancy, we rethink graph pooling and leverage it to automatically generate multi-scale positive views.
  arXiv  Detail & Related papers  (2024-01-29T10:00:53Z)
• Hierarchical Contrastive Learning Enhanced Heterogeneous Graph Neural Network [59.860534520941485]
  Heterogeneous graph neural networks (HGNNs) as an emerging technique have shown superior capacity of dealing with heterogeneous information network (HIN) Recently, contrastive learning, a self-supervised method, becomes one of the most exciting learning paradigms and shows great potential when there are no labels. In this paper, we study the problem of self-supervised HGNNs and propose a novel co-contrastive learning mechanism for HGNNs, named HeCo.
  arXiv  Detail & Related papers  (2023-04-24T16:17:21Z)
• On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning [8.959800369169798]
  It is not well understood why contrastive learning works for learning sentence-level semantics. We show that contrastive learning brings isotropy, and drives high intra-sentence similarity. We also find that what we formalize as "spurious contextualization" is mitigated for semantically meaningful tokens.
  arXiv  Detail & Related papers  (2022-12-18T21:11:49Z)
• Contrastive Learning with Boosted Memorization [36.957895270908324]
  Self-supervised learning has achieved a great success in the representation learning of visual and textual data. Recent attempts to consider self-supervised long-tailed learning are made by rebalancing in the loss perspective or the model perspective. We propose a novel Boosted Contrastive Learning (BCL) method to enhance the long-tailed learning in the label-unaware context.
  arXiv  Detail & Related papers  (2022-05-25T11:54:22Z)
• Unified Contrastive Learning in Image-Text-Label Space [130.31947133453406]
  Unified Contrastive Learning (UniCL) is effective way of learning semantically rich yet discriminative representations. UniCL stand-alone is a good learner on pure imagelabel data, rivaling the supervised learning methods across three image classification datasets.
  arXiv  Detail & Related papers  (2022-04-07T17:34:51Z)
• A Simple Long-Tailed Recognition Baseline via Vision-Language Model [92.2866546058082]
  The visual world naturally exhibits a long-tailed distribution of open classes, which poses great challenges to modern visual systems. Recent advances in contrastive visual-language pretraining shed light on a new pathway for visual recognition. We propose BALLAD to leverage contrastive vision-language models for long-tailed recognition.
  arXiv  Detail & Related papers  (2021-11-29T17:49:24Z)
• Semi-Supervising Learning, Transfer Learning, and Knowledge Distillation with SimCLR [2.578242050187029]
  Recent breakthroughs in the field of semi-supervised learning have achieved results that match state-of-the-art traditional supervised learning methods. SimCLR is the current state-of-the-art semi-supervised learning framework for computer vision.
  arXiv  Detail & Related papers  (2021-08-02T01:37:39Z)
• CoCon: Cooperative-Contrastive Learning [52.342936645996765]
  Self-supervised visual representation learning is key for efficient video analysis. Recent success in learning image representations suggests contrastive learning is a promising framework to tackle this challenge. We introduce a cooperative variant of contrastive learning to utilize complementary information across views.
  arXiv  Detail & Related papers  (2021-04-30T05:46:02Z)
• Hard Negative Mixing for Contrastive Learning [29.91220669060252]
  We argue that an important aspect of contrastive learning, i.e., the effect of hard negatives, has so far been neglected. We propose hard negative mixing strategies at the feature level, that can be computed on-the-fly with a minimal computational overhead.
  arXiv  Detail & Related papers  (2020-10-02T14:34:58Z)
• Revisiting Meta-Learning as Supervised Learning [69.2067288158133]
  We aim to provide a principled, unifying framework by revisiting and strengthening the connection between meta-learning and traditional supervised learning. By treating pairs of task-specific data sets and target models as (feature, label) samples, we can reduce many meta-learning algorithms to instances of supervised learning. This view not only unifies meta-learning into an intuitive and practical framework but also allows us to transfer insights from supervised learning directly to improve meta-learning.
  arXiv  Detail & Related papers  (2020-02-03T06:13:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.