The Power of Contrast for Feature Learning: A Theoretical Analysis
- URL: http://arxiv.org/abs/2110.02473v4
- Date: Wed, 20 Dec 2023 03:59:21 GMT
- Title: The Power of Contrast for Feature Learning: A Theoretical Analysis
- Authors: Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, Linjun Zhang
- Abstract summary: We show that contrastive learning outperforms the standard autoencoders and generative adversarial networks.
We also illustrate the impact of labeled data in supervised contrastive learning.
- Score: 42.20116348668721
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Contrastive learning has achieved state-of-the-art performance in various
self-supervised learning tasks and even outperforms its supervised counterpart.
Despite its empirical success, theoretical understanding of the superiority of
contrastive learning is still limited. In this paper, under linear
representation settings, (i) we provably show that contrastive learning
outperforms the standard autoencoders and generative adversarial networks, two
classical generative unsupervised learning methods, for both feature recovery
and in-domain downstream tasks; (ii) we also illustrate the impact of labeled
data in supervised contrastive learning. This provides theoretical support for
recent findings that contrastive learning with labels improves the performance
of learned representations in the in-domain downstream task, but it can harm
the performance in transfer learning. We verify our theory with numerical
experiments.
Related papers
- On the Comparison between Multi-modal and Single-modal Contrastive Learning [50.74988548106031]
We introduce a theoretical foundation for understanding the differences between multi-modal and single-modal contrastive learning.
We identify the critical factor, which is the signal-to-noise ratio (SNR), that impacts the generalizability in downstream tasks of both multi-modal and single-modal contrastive learning.
Our analysis provides a unified framework that can characterize the optimization and generalization of both single-modal and multi-modal contrastive learning.
arXiv Detail & Related papers (2024-11-05T06:21:17Z) - Provable Contrastive Continual Learning [7.6989463205452555]
We establish theoretical performance guarantees, which reveal how the performance of the model is bounded by training losses of previous tasks.
Inspired by our theoretical analysis of these guarantees, we propose a novel contrastive continual learning algorithm called CILA.
Our method shows great improvement on standard benchmarks and achieves new state-of-the-art performance.
arXiv Detail & Related papers (2024-05-29T04:48:11Z) - ArCL: Enhancing Contrastive Learning with Augmentation-Robust
Representations [30.745749133759304]
We develop a theoretical framework to analyze the transferability of self-supervised contrastive learning.
We show that contrastive learning fails to learn domain-invariant features, which limits its transferability.
Based on these theoretical insights, we propose a novel method called Augmentation-robust Contrastive Learning (ArCL)
arXiv Detail & Related papers (2023-03-02T09:26:20Z) - A Theoretical Study of Inductive Biases in Contrastive Learning [32.98250585760665]
We provide the first theoretical analysis of self-supervised learning that incorporates the effect of inductive biases originating from the model class.
We show that when the model has limited capacity, contrastive representations would recover certain special clustering structures that are compatible with the model architecture.
arXiv Detail & Related papers (2022-11-27T01:53:29Z) - Conditional Supervised Contrastive Learning for Fair Text Classification [59.813422435604025]
We study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning.
Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives.
arXiv Detail & Related papers (2022-05-23T17:38:30Z) - Understanding Contrastive Learning Requires Incorporating Inductive
Biases [64.56006519908213]
Recent attempts to theoretically explain the success of contrastive learning on downstream tasks prove guarantees depending on properties of em augmentations and the value of em contrastive loss of representations.
We demonstrate that such analyses ignore em inductive biases of the function class and training algorithm, even em provably leading to vacuous guarantees in some settings.
arXiv Detail & Related papers (2022-02-28T18:59:20Z) - Co$^2$L: Contrastive Continual Learning [69.46643497220586]
Recent breakthroughs in self-supervised learning show that such algorithms learn visual representations that can be transferred better to unseen tasks.
We propose a rehearsal-based continual learning algorithm that focuses on continually learning and maintaining transferable representations.
arXiv Detail & Related papers (2021-06-28T06:14:38Z) - Hybrid Discriminative-Generative Training via Contrastive Learning [96.56164427726203]
We show that through the perspective of hybrid discriminative-generative training of energy-based models we can make a direct connection between contrastive learning and supervised learning.
We show our specific choice of approximation of the energy-based loss outperforms the existing practice in terms of classification accuracy of WideResNet on CIFAR-10 and CIFAR-100.
arXiv Detail & Related papers (2020-07-17T15:50:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.