Modulate Your Spectrum in Self-Supervised Learning

• URL: http://arxiv.org/abs/2305.16789v2
• Date: Mon, 22 Jan 2024 02:47:50 GMT
• Title: Modulate Your Spectrum in Self-Supervised Learning
• Authors: Xi Weng, Yunhao Ni, Tengwei Song, Jie Luo, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan, Lei Huang
• Abstract summary: Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning. We introduce Spectral Transformation (ST), a framework to modulate the spectrum of embedding. We propose a novel ST instance named IterNorm with trace loss (INTL)
• Score: 65.963806450552
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning (SSL) with joint embedding architectures. Typically, it involves a hard whitening approach, transforming the embedding and applying loss to the whitened output. In this work, we introduce Spectral Transformation (ST), a framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. We show that whitening is a special instance of ST by definition, and our empirical investigations unveil other ST instances capable of preventing collapse. Additionally, we propose a novel ST instance named IterNorm with trace loss (INTL). Theoretical analysis confirms INTL's efficacy in preventing collapse and modulating the spectrum of embedding toward equal-eigenvalues during optimization. Our experiments on ImageNet classification and COCO object detection demonstrate INTL's potential in learning superior representations. The code is available at https://github.com/winci-ai/INTL.

Related papers

• Whitening Consistently Improves Self-Supervised Learning [5.0337106694127725]
  We propose incorporating ZCA whitening as the final layer of the encoder in self-supervised learning. Our experiments show that whitening is capable of improving linear and k-NN probing accuracy by 1-5%.
  arXiv  Detail & Related papers  (2024-08-14T12:52:13Z)
• Breaking Secure Aggregation: Label Leakage from Aggregated Gradients in Federated Learning [11.18348760596715]
  Federated Learning exhibits privacy vulnerabilities under gradient inversion attacks (GIAs) We propose a stealthy label inference attack to bypass Secure Aggregation (SA) and recover individual clients' private labels. Our attack achieves large-scale label recovery with 100% accuracy on various datasets and model architectures.
  arXiv  Detail & Related papers  (2024-06-22T04:42:18Z)
• Balancing Embedding Spectrum for Recommendation [7.523823738965443]
  We show that representations tend to span a subspace of the whole embedding space, leading to a suboptimal solution and reducing the model capacity. We propose a novel method called DirectSpec to balance the spectrum distribution of the embeddings during training. We also propose an enhanced variant, DirectSpec+, which employs self-paced gradients to optimize irrelevant samples more effectively.
  arXiv  Detail & Related papers  (2024-06-17T18:59:43Z)
• Triplet Contrastive Learning for Unsupervised Vehicle Re-identification [55.445358749042384]
  Part feature learning is a critical technology for fine semantic understanding in vehicle re-identification. We propose a novel Triplet Contrastive Learning framework (TCL) which leverages cluster features to bridge the part features and global features.
  arXiv  Detail & Related papers  (2023-01-23T15:52:12Z)
• An Investigation into Whitening Loss for Self-supervised Learning [62.157102463386394]
  A desirable objective in self-supervised learning (SSL) is to avoid feature collapse. We propose a framework with an informative indicator to analyze whitening loss. Based on our analysis, we propose channel whitening with random group partition (CW-RGP)
  arXiv  Detail & Related papers  (2022-10-07T14:43:29Z)
• Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap [64.60460828425502]
  We propose a new guarantee on the downstream performance of contrastive learning. Our new theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations. We propose an unsupervised model selection metric ARC that aligns well with downstream accuracy.
  arXiv  Detail & Related papers  (2022-03-25T05:36:26Z)
• Understanding Self-supervised Learning with Dual Deep Networks [74.92916579635336]
  We propose a novel framework to understand contrastive self-supervised learning (SSL) methods that employ dual pairs of deep ReLU networks. We prove that in each SGD update of SimCLR with various loss functions, the weights at each layer are updated by a emphcovariance operator. To further study what role the covariance operator plays and which features are learned in such a process, we model data generation and augmentation processes through a emphhierarchical latent tree model (HLTM)
  arXiv  Detail & Related papers  (2020-10-01T17:51:49Z)
• Whitening for Self-Supervised Representation Learning [129.57407186848917]
  We propose a new loss function for self-supervised representation learning (SSL) based on the whitening of latent-space features. Our solution does not require asymmetric networks and it is conceptually simple.
  arXiv  Detail & Related papers  (2020-07-13T12:33:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.