Contrastive learning, multi-view redundancy, and linear models

• URL: http://arxiv.org/abs/2008.10150v2
• Date: Wed, 14 Apr 2021 19:19:55 GMT
• Title: Contrastive learning, multi-view redundancy, and linear models
• Authors: Christopher Tosh, Akshay Krishnamurthy, Daniel Hsu
• Abstract summary: A popular self-supervised approach to representation learning is contrastive learning. This work provides a theoretical analysis of contrastive learning in the multi-view setting.
• Score: 38.80336134485453
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised learning is an empirically successful approach to unsupervised learning based on creating artificial supervised learning problems. A popular self-supervised approach to representation learning is contrastive learning, which leverages naturally occurring pairs of similar and dissimilar data points, or multiple views of the same data. This work provides a theoretical analysis of contrastive learning in the multi-view setting, where two views of each datum are available. The main result is that linear functions of the learned representations are nearly optimal on downstream prediction tasks whenever the two views provide redundant information about the label.

Related papers

• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning [32.15608637930748]
  We show that there exists a trade-off between the two desiderata so that one may not be able to achieve both simultaneously. We provide analysis using a theoretical data model and show that, while more diverse pre-training data result in more diverse features for different tasks, it puts less emphasis on task-specific features.
  arXiv  Detail & Related papers  (2023-02-28T22:14:33Z)
• Understanding Self-Predictive Learning for Reinforcement Learning [61.62067048348786]
  We study the learning dynamics of self-predictive learning for reinforcement learning. We propose a novel self-predictive algorithm that learns two representations simultaneously.
  arXiv  Detail & Related papers  (2022-12-06T20:43:37Z)
• Cross-view Graph Contrastive Representation Learning on Partially Aligned Multi-view Data [52.491074276133325]
  Multi-view representation learning has developed rapidly over the past decades and has been applied in many fields. We propose a new cross-view graph contrastive learning framework, which integrates multi-view information to align data and learn latent representations. Experiments conducted on several real datasets demonstrate the effectiveness of the proposed method on the clustering and classification tasks.
  arXiv  Detail & Related papers  (2022-11-08T09:19:32Z)
• Co$^2$L: Contrastive Continual Learning [69.46643497220586]
  Recent breakthroughs in self-supervised learning show that such algorithms learn visual representations that can be transferred better to unseen tasks. We propose a rehearsal-based continual learning algorithm that focuses on continually learning and maintaining transferable representations.
  arXiv  Detail & Related papers  (2021-06-28T06:14:38Z)
• Distill on the Go: Online knowledge distillation in self-supervised learning [1.1470070927586016]
  Recent works have shown that wider and deeper models benefit more from self-supervised learning than smaller models. We propose Distill-on-the-Go (DoGo), a self-supervised learning paradigm using single-stage online knowledge distillation. Our results show significant performance gain in the presence of noisy and limited labels.
  arXiv  Detail & Related papers  (2021-04-20T09:59:23Z)
• Self-supervised Learning from a Multi-view Perspective [121.63655399591681]
  We show that self-supervised representations can extract task-relevant information and discard task-irrelevant information. Our theoretical framework paves the way to a larger space of self-supervised learning objective design.
  arXiv  Detail & Related papers  (2020-06-10T00:21:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.