Understanding Dimensional Collapse in Contrastive Self-supervised Learning

• URL: http://arxiv.org/abs/2110.09348v1
• Date: Mon, 18 Oct 2021 14:22:19 GMT
• Title: Understanding Dimensional Collapse in Contrastive Self-supervised Learning
• Authors: Li Jing, Pascal Vincent, Yann LeCun, Yuandong Tian
• Abstract summary: We show that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse. Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR, which directly optimize the representation space without relying on a trainable projector.
• Score: 57.98014222570084
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Self-supervised visual representation learning aims to learn useful representations without relying on human annotations. Joint embedding approach bases on maximizing the agreement between embedding vectors from different views of the same image. Various methods have been proposed to solve the collapsing problem where all embedding vectors collapse to a trivial constant solution. Among these methods, contrastive learning prevents collapse via negative sample pairs. It has been shown that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse, whereby the embedding vectors end up spanning a lower-dimensional subspace instead of the entire available embedding space. Here, we show that dimensional collapse also happens in contrastive learning. In this paper, we shed light on the dynamics at play in contrastive learning that leads to dimensional collapse. Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR, which directly optimizes the representation space without relying on a trainable projector. Experiments show that DirectCLR outperforms SimCLR with a trainable linear projector on ImageNet.

Related papers

• Contrastive Self-Supervised Learning As Neural Manifold Packing [0.0]
  We introduce Contrastive Learning As Manifold Packing (CLAMP), a self-supervised framework that recasts representation learning as a manifold packing problem.<n>In this framework, each class consists of sub-manifolds embedding multiple augmented views of a single image.<n>Under the standard linear evaluation protocol, CLAMP achieves competitive performance with state-of-the-art self-supervised models.
  arXiv  Detail & Related papers  (2025-06-16T17:24:31Z)
• Machine Unlearning in Hyperbolic vs. Euclidean Multimodal Contrastive Learning: Adapting Alignment Calibration to MERU [50.9588132578029]
  This paper investigates machine unlearning in hyperbolic contrastive learning. We adapt Alignment to MERU, a model that embeds images and text in hyperbolic space to better capture semantic hierarchies. Our approach introduces hyperbolic-specific components including entailment calibration and norm regularization that leverage the unique properties of hyperbolic space.
  arXiv  Detail & Related papers  (2025-03-19T12:47:37Z)
• Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP) [0.0]
  CLOP is a novel semi-supervised loss function designed to prevent neural collapse by promoting the formation of linear subspaces among class embeddings. We show that CLOP enhances performance, providing greater stability across different learning rates and batch sizes.
  arXiv  Detail & Related papers  (2024-03-27T15:48:16Z)
• Alignment and Outer Shell Isotropy for Hyperbolic Graph Contrastive Learning [69.6810940330906]
  We propose a novel contrastive learning framework to learn high-quality graph embedding. Specifically, we design the alignment metric that effectively captures the hierarchical data-invariant information. We show that in the hyperbolic space one has to address the leaf- and height-level uniformity which are related to properties of trees.
  arXiv  Detail & Related papers  (2023-10-27T15:31:42Z)
• Preventing Dimensional Collapse of Incomplete Multi-View Clustering via Direct Contrastive Learning [0.14999444543328289]
  We propose a novel incomplete multi-view contrastive clustering framework. It effectively avoids dimensional collapse without relying on projection heads. It achieves state-of-the-art clustering results on 5 public datasets.
  arXiv  Detail & Related papers  (2023-03-22T00:21:50Z)
• ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond [13.888935924826903]
  Oversmoothing is a common phenomenon in a wide range of Graph Neural Networks (GNNs) and Transformers. We propose a novel normalization layer called ContraNorm, which implicitly shatters representations in the embedding space. Our proposed normalization layer can be easily integrated into GNNs and Transformers with negligible parameter overhead.
  arXiv  Detail & Related papers  (2023-03-12T04:04:51Z)
• Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap [64.60460828425502]
  We propose a new guarantee on the downstream performance of contrastive learning. Our new theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations. We propose an unsupervised model selection metric ARC that aligns well with downstream accuracy.
  arXiv  Detail & Related papers  (2022-03-25T05:36:26Z)
• Towards Demystifying Representation Learning with Non-contrastive Self-supervision [82.80118139087676]
  Non-contrastive methods of self-supervised learning learn representations by minimizing the distance between two views of the same image. Tian el al. (2021) made an initial attempt on the first question and proposed DirectPred that sets the predictor directly. We show that in a simple linear network, DirectSet($alpha$) provably learns a desirable projection matrix and also reduces the sample complexity on downstream tasks.
  arXiv  Detail & Related papers  (2021-10-11T00:48:05Z)
• Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation [64.92152574895111]
  We propose a simple Orthogonal Jacobian Regularization (OroJaR) to encourage deep generative model to learn disentangled representations. Our method is effective in disentangled and controllable image generation, and performs favorably against the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-17T15:01:46Z)
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning [43.96465407127458]
  We introduce VICReg, a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings. VICReg achieves results on par with the state of the art on several downstream tasks.
  arXiv  Detail & Related papers  (2021-05-11T09:53:21Z)
• Understanding self-supervised Learning Dynamics without Contrastive Pairs [72.1743263777693]
  Contrastive approaches to self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point. BYOL and SimSiam, show remarkable performance it without negative pairs. We study the nonlinear learning dynamics of non-contrastive SSL in simple linear networks.
  arXiv  Detail & Related papers  (2021-02-12T22:57:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.