An Empirical Study on Disentanglement of Negative-free Contrastive Learning

• URL: http://arxiv.org/abs/2206.04756v1
• Date: Thu, 9 Jun 2022 20:25:34 GMT
• Title: An Empirical Study on Disentanglement of Negative-free Contrastive Learning
• Authors: Jinkun Cao, Ruiqian Nai, Qing Yang, Jialei Huang, Yang Gao
• Abstract summary: We take different negative-free contrastive learning methods to study the disentanglement property of this genre of self-supervised methods. We propose a new disentanglement metric based on Mutual Information between representation and data factors. Our study shows that the investigated methods can learn a well-disentangled subset of representation.
• Score: 9.52192693120765
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Negative-free contrastive learning has attracted a lot of attention with simplicity and impressive performance for large-scale pretraining. But its disentanglement property remains unexplored. In this paper, we take different negative-free contrastive learning methods to study the disentanglement property of this genre of self-supervised methods empirically. We find the existing disentanglement metrics fail to make meaningful measurements for the high-dimensional representation model so we propose a new disentanglement metric based on Mutual Information between representation and data factors. With the proposed metric, we benchmark the disentanglement property of negative-free contrastive learning for the first time, on both popular synthetic datasets and a real-world dataset CelebA. Our study shows that the investigated methods can learn a well-disentangled subset of representation. We extend the study of the disentangled representation learning to high-dimensional representation space and negative-free contrastive learning for the first time. The implementation of the proposed metric is available at \url{https://github.com/noahcao/disentanglement_lib_med}.

Related papers

• An Attention-based Framework for Fair Contrastive Learning [2.1605931466490795]
  We propose a new method for fair contrastive learning that employs an attention mechanism to model bias-causing interactions. Our attention mechanism avoids bias-causing samples that confound the model and focuses on bias-reducing samples that help learn semantically meaningful representations.
  arXiv  Detail & Related papers  (2024-11-22T07:11:35Z)
• Understanding Collapse in Non-Contrastive Learning [122.2499276246997]
  We show that SimSiam representations undergo partial dimensional collapse if the model is too small relative to the dataset size. We propose a metric to measure the degree of this collapse and show that it can be used to forecast the downstream task performance without any fine-tuning or labels.
  arXiv  Detail & Related papers  (2022-09-29T17:59:55Z)
• Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization [77.8735802150511]
  We propose a cross-sample adversarial debiasing (CSAD) method to remove the bias information misused by the target task. The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator. We conduct thorough experiments on publicly available datasets to validate the advantages of the proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-08-11T21:17:02Z)
• Unsupervised Embedding Learning from Uncertainty Momentum Modeling [37.674449317054716]
  We propose a novel solution to explicitly model and explore the uncertainty of the given unlabeled learning samples. We leverage such uncertainty modeling momentum to the learning which is helpful to tackle the outliers.
  arXiv  Detail & Related papers  (2021-07-19T14:06:19Z)
• Investigating the Role of Negatives in Contrastive Representation Learning [59.30700308648194]
  Noise contrastive learning is a popular technique for unsupervised representation learning. We focus on disambiguating the role of one of these parameters: the number of negative examples. We find that the results broadly agree with our theory, while our vision experiments are murkier with performance sometimes even being insensitive to the number of negatives.
  arXiv  Detail & Related papers  (2021-06-18T06:44:16Z)
• Incremental False Negative Detection for Contrastive Learning [95.68120675114878]
  We introduce a novel incremental false negative detection for self-supervised contrastive learning. During contrastive learning, we discuss two strategies to explicitly remove the detected false negatives. Our proposed method outperforms other self-supervised contrastive learning frameworks on multiple benchmarks within a limited compute.
  arXiv  Detail & Related papers  (2021-06-07T15:29:14Z)
• A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation [63.042651834453544]
  We show that the unsupervised learning of disentangled representations is impossible without inductive biases on both the models and the data. We observe that while the different methods successfully enforce properties "encouraged" by the corresponding losses, well-disentangled models seemingly cannot be identified without supervision. Our results suggest that future work on disentanglement learning should be explicit about the role of inductive biases and (implicit) supervision.
  arXiv  Detail & Related papers  (2020-10-27T10:17:15Z)
• Deep Dimension Reduction for Supervised Representation Learning [51.10448064423656]
  We propose a deep dimension reduction approach to learning representations with essential characteristics. The proposed approach is a nonparametric generalization of the sufficient dimension reduction method. We show that the estimated deep nonparametric representation is consistent in the sense that its excess risk converges to zero.
  arXiv  Detail & Related papers  (2020-06-10T14:47:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.