Understanding Dimensional Collapse in Contrastive Self-supervised
Learning
- URL: http://arxiv.org/abs/2110.09348v1
- Date: Mon, 18 Oct 2021 14:22:19 GMT
- Title: Understanding Dimensional Collapse in Contrastive Self-supervised
Learning
- Authors: Li Jing, Pascal Vincent, Yann LeCun, Yuandong Tian
- Abstract summary: We show that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse.
Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR, which directly optimize the representation space without relying on a trainable projector.
- Score: 57.98014222570084
- License: http://creativecommons.org/publicdomain/zero/1.0/
- Abstract: Self-supervised visual representation learning aims to learn useful
representations without relying on human annotations. Joint embedding approach
bases on maximizing the agreement between embedding vectors from different
views of the same image. Various methods have been proposed to solve the
collapsing problem where all embedding vectors collapse to a trivial constant
solution. Among these methods, contrastive learning prevents collapse via
negative sample pairs. It has been shown that non-contrastive methods suffer
from a lesser collapse problem of a different nature: dimensional collapse,
whereby the embedding vectors end up spanning a lower-dimensional subspace
instead of the entire available embedding space. Here, we show that dimensional
collapse also happens in contrastive learning. In this paper, we shed light on
the dynamics at play in contrastive learning that leads to dimensional
collapse. Inspired by our theory, we propose a novel contrastive learning
method, called DirectCLR, which directly optimizes the representation space
without relying on a trainable projector. Experiments show that DirectCLR
outperforms SimCLR with a trainable linear projector on ImageNet.
Related papers
- Alignment and Outer Shell Isotropy for Hyperbolic Graph Contrastive
Learning [69.6810940330906]
We propose a novel contrastive learning framework to learn high-quality graph embedding.
Specifically, we design the alignment metric that effectively captures the hierarchical data-invariant information.
We show that in the hyperbolic space one has to address the leaf- and height-level uniformity which are related to properties of trees.
arXiv Detail & Related papers (2023-10-27T15:31:42Z) - Preventing Dimensional Collapse of Incomplete Multi-View Clustering via
Direct Contrastive Learning [0.14999444543328289]
We propose a novel incomplete multi-view contrastive clustering framework.
It effectively avoids dimensional collapse without relying on projection heads.
It achieves state-of-the-art clustering results on 5 public datasets.
arXiv Detail & Related papers (2023-03-22T00:21:50Z) - ContraNorm: A Contrastive Learning Perspective on Oversmoothing and
Beyond [13.888935924826903]
Oversmoothing is a common phenomenon in a wide range of Graph Neural Networks (GNNs) and Transformers.
We propose a novel normalization layer called ContraNorm, which implicitly shatters representations in the embedding space.
Our proposed normalization layer can be easily integrated into GNNs and Transformers with negligible parameter overhead.
arXiv Detail & Related papers (2023-03-12T04:04:51Z) - Linear Adversarial Concept Erasure [98.14246446690282]
We formulate the problem of identifying and erasing a linear subspace that corresponds to a given concept.
We model this problem as a constrained, linear minimax game, and show that existing solutions are generally not optimal for this task.
We show that the method -- despite being linear -- is highly expressive, effectively mitigating bias in deep nonlinear classifiers while maintaining tractability and interpretability.
arXiv Detail & Related papers (2022-01-28T13:00:17Z) - Towards Demystifying Representation Learning with Non-contrastive
Self-supervision [82.80118139087676]
Non-contrastive methods of self-supervised learning learn representations by minimizing the distance between two views of the same image.
Tian el al. (2021) made an initial attempt on the first question and proposed DirectPred that sets the predictor directly.
We show that in a simple linear network, DirectSet($alpha$) provably learns a desirable projection matrix and also reduces the sample complexity on downstream tasks.
arXiv Detail & Related papers (2021-10-11T00:48:05Z) - Orthogonal Jacobian Regularization for Unsupervised Disentanglement in
Image Generation [64.92152574895111]
We propose a simple Orthogonal Jacobian Regularization (OroJaR) to encourage deep generative model to learn disentangled representations.
Our method is effective in disentangled and controllable image generation, and performs favorably against the state-of-the-art methods.
arXiv Detail & Related papers (2021-08-17T15:01:46Z) - VICReg: Variance-Invariance-Covariance Regularization for
Self-Supervised Learning [43.96465407127458]
We introduce VICReg, a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings.
VICReg achieves results on par with the state of the art on several downstream tasks.
arXiv Detail & Related papers (2021-05-11T09:53:21Z) - Understanding self-supervised Learning Dynamics without Contrastive
Pairs [72.1743263777693]
Contrastive approaches to self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point.
BYOL and SimSiam, show remarkable performance it without negative pairs.
We study the nonlinear learning dynamics of non-contrastive SSL in simple linear networks.
arXiv Detail & Related papers (2021-02-12T22:57:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.