An Investigation into Whitening Loss for Self-supervised Learning

• URL: http://arxiv.org/abs/2210.03586v1
• Date: Fri, 7 Oct 2022 14:43:29 GMT
• Title: An Investigation into Whitening Loss for Self-supervised Learning
• Authors: Xi Weng, Lei Huang, Lei Zhao, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan
• Abstract summary: A desirable objective in self-supervised learning (SSL) is to avoid feature collapse. We propose a framework with an informative indicator to analyze whitening loss. Based on our analysis, we propose channel whitening with random group partition (CW-RGP)
• Score: 62.157102463386394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A desirable objective in self-supervised learning (SSL) is to avoid feature collapse. Whitening loss guarantees collapse avoidance by minimizing the distance between embeddings of positive pairs under the conditioning that the embeddings from different views are whitened. In this paper, we propose a framework with an informative indicator to analyze whitening loss, which provides a clue to demystify several interesting phenomena as well as a pivoting point connecting to other SSL methods. We reveal that batch whitening (BW) based methods do not impose whitening constraints on the embedding, but they only require the embedding to be full-rank. This full-rank constraint is also sufficient to avoid dimensional collapse. Based on our analysis, we propose channel whitening with random group partition (CW-RGP), which exploits the advantages of BW-based methods in preventing collapse and avoids their disadvantages requiring large batch size. Experimental results on ImageNet classification and COCO object detection reveal that the proposed CW-RGP possesses a promising potential for learning good representations. The code is available at https://github.com/winci-ai/CW-RGP.

Related papers

• Covariance-corrected Whitening Alleviates Network Degeneration on Imbalanced Classification [6.197116272789107]
  Class imbalance is a critical issue in image classification that significantly affects the performance of deep recognition models. We propose a novel framework called Whitening-Net to mitigate the degenerate solutions. In scenarios with extreme class imbalance, the batch covariance statistic exhibits significant fluctuations, impeding the convergence of the whitening operation.
  arXiv  Detail & Related papers  (2024-08-30T10:49:33Z)
• Whitening Not Recommended for Classification Tasks in LLMs [0.08192907805418582]
  Whitening has been claimed to be an effective operation to improve embedding quality obtained from Large Language Models (LLMs) In particular, whitening degenerates embeddings for classification tasks. A by-product of our research is embedding evaluation platform for LLMs called SentEval+.
  arXiv  Detail & Related papers  (2024-07-16T22:48:30Z)
• Whitening-based Contrastive Learning of Sentence Embeddings [61.38955786965527]
  This paper presents a whitening-based contrastive learning method for sentence embedding learning (WhitenedCSE) We find that these two approaches are not totally redundant but actually have some complementarity due to different uniformity mechanism.
  arXiv  Detail & Related papers  (2023-05-28T14:58:10Z)
• Modulate Your Spectrum in Self-Supervised Learning [65.963806450552]
  Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning. We introduce Spectral Transformation (ST), a framework to modulate the spectrum of embedding. We propose a novel ST instance named IterNorm with trace loss (INTL)
  arXiv  Detail & Related papers  (2023-05-26T09:59:48Z)
• Triplet Contrastive Learning for Unsupervised Vehicle Re-identification [55.445358749042384]
  Part feature learning is a critical technology for fine semantic understanding in vehicle re-identification. We propose a novel Triplet Contrastive Learning framework (TCL) which leverages cluster features to bridge the part features and global features.
  arXiv  Detail & Related papers  (2023-01-23T15:52:12Z)
• Improving Generalization of Batch Whitening by Convolutional Unit Optimization [24.102442375834084]
  Batch Whitening is a technique that accelerates and stabilizes training by transforming input features to have a zero mean (Centering) and a unit variance (Scaling) In commonly used structures, which are empirically optimized with Batch Normalization, the normalization layer appears between convolution and activation function. We propose a new Convolutional Unit that is in line with the theory, and our method generally improves the performance of Batch Whitening.
  arXiv  Detail & Related papers  (2021-08-24T10:27:57Z)
• Boosting Gradient for White-Box Adversarial Attacks [60.422511092730026]
  We propose a universal adversarial example generation method, called ADV-ReLU, to enhance the performance of gradient based white-box attack algorithms. Our approach calculates the gradient of the loss function versus network input, maps the values to scores, and selects a part of them to update the misleading gradients.
  arXiv  Detail & Related papers  (2020-10-21T02:13:26Z)
• Whitening for Self-Supervised Representation Learning [129.57407186848917]
  We propose a new loss function for self-supervised representation learning (SSL) based on the whitening of latent-space features. Our solution does not require asymmetric networks and it is conceptually simple.
  arXiv  Detail & Related papers  (2020-07-13T12:33:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.