VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

• URL: http://arxiv.org/abs/2105.04906v1
• Date: Tue, 11 May 2021 09:53:21 GMT
• Title: VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
• Authors: Adrien Bardes and Jean Ponce and Yann LeCun
• Abstract summary: We introduce VICReg, a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings. VICReg achieves results on par with the state of the art on several downstream tasks.
• Score: 43.96465407127458
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image. A trivial solution is obtained when the encoder outputs constant vectors. This collapse problem is often avoided through implicit biases in the learning architecture, that often lack a clear justification or interpretation. In this paper, we introduce VICReg (Variance-Invariance-Covariance Regularization), a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings along each dimension individually. VICReg combines the variance term with a decorrelation mechanism based on redundancy reduction and covariance regularization, and achieves results on par with the state of the art on several downstream tasks. In addition, we show that incorporating our new variance term into other methods helps stabilize the training and leads to performance improvements.

Related papers

• The Benefits of Balance: From Information Projections to Variance Reduction [7.082773426322819]
  We show that an iterative algorithm, usually used to avoid representation collapse, enjoys an unsuspected benefit. We provide non-asymptotic bounds quantifying this variance reduction effect and relate them to the eigendecays of appropriately defined Markov operators. We explain how various forms of data balancing in contrastive multimodal learning and self-supervised clustering can be interpreted as instances of this variance reduction scheme.
  arXiv  Detail & Related papers  (2024-08-27T13:48:15Z)
• Unsupervised Representation Learning by Balanced Self Attention Matching [2.3020018305241337]
  We present a self-supervised method for embedding image features called BAM. We obtain rich representations and avoid feature collapse by minimizing a loss that matches these distributions to their globally balanced and entropy regularized version. We show competitive performance with leading methods on both semi-supervised and transfer-learning benchmarks.
  arXiv  Detail & Related papers  (2024-08-04T12:52:44Z)
• ER: Equivariance Regularizer for Knowledge Graph Completion [107.51609402963072]
  We propose a new regularizer, namely, Equivariance Regularizer (ER) ER can enhance the generalization ability of the model by employing the semantic equivariance between the head and tail entities. The experimental results indicate a clear and substantial improvement over the state-of-the-art relation prediction methods.
  arXiv  Detail & Related papers  (2022-06-24T08:18:05Z)
• TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning [9.507070656654632]
  We present Transformation Invariance and Covariance Contrast (TiCo) for self-supervised visual representation learning. Our method is based on maximizing the agreement among embeddings of different distorted versions of the same image. We show that TiCo can be viewed as a variant of MoCo with an implicit memory bank of unlimited size at no extra memory cost.
  arXiv  Detail & Related papers  (2022-06-21T19:44:01Z)
• Regularising for invariance to data augmentation improves supervised learning [82.85692486314949]
  We show that using multiple augmentations per input can improve generalisation. We propose an explicit regulariser that encourages this invariance on the level of individual model predictions.
  arXiv  Detail & Related papers  (2022-03-07T11:25:45Z)
• Revisiting Consistency Regularization for Semi-Supervised Learning [80.28461584135967]
  We propose an improved consistency regularization framework by a simple yet effective technique, FeatDistLoss. Experimental results show that our model defines a new state of the art for various datasets and settings.
  arXiv  Detail & Related papers  (2021-12-10T20:46:13Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• Generalized Negative Correlation Learning for Deep Ensembling [7.569288952340753]
  Ensemble algorithms offer state of the art performance in many machine learning applications. We formulate a generalized bias-variance decomposition for arbitrary twice differentiable loss functions. We derive a Generalized Negative Correlation Learning algorithm which offers explicit control over the ensemble's diversity.
  arXiv  Detail & Related papers  (2020-11-05T16:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.