Correlation between Alignment-Uniformity and Performance of Dense Contrastive Representations

• URL: http://arxiv.org/abs/2210.08819v1
• Date: Mon, 17 Oct 2022 08:08:37 GMT
• Title: Correlation between Alignment-Uniformity and Performance of Dense Contrastive Representations
• Authors: Jong Hak Moon, Wonjae Kim, and Edward Choi
• Abstract summary: We analyze the theoretical ideas of dense contrastive learning using a standard CNN and straightforward feature matching scheme. We discover the core principle in constructing a positive pair of dense features and empirically proved its validity. Also, we introduce a new scalar metric that summarizes the correlation between alignment-and-uniformity and downstream performance.
• Score: 11.266613717084788
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, dense contrastive learning has shown superior performance on dense prediction tasks compared to instance-level contrastive learning. Despite its supremacy, the properties of dense contrastive representations have not yet been carefully studied. Therefore, we analyze the theoretical ideas of dense contrastive learning using a standard CNN and straightforward feature matching scheme rather than propose a new complex method. Inspired by the analysis of the properties of instance-level contrastive representations through the lens of alignment and uniformity on the hypersphere, we employ and extend the same lens for the dense contrastive representations to analyze their underexplored properties. We discover the core principle in constructing a positive pair of dense features and empirically proved its validity. Also, we introduces a new scalar metric that summarizes the correlation between alignment-and-uniformity and downstream performance. Using this metric, we study various facets of densely learned contrastive representations such as how the correlation changes over single- and multi-object datasets or linear evaluation and dense prediction tasks. The source code is publicly available at: https://github.com/SuperSupermoon/DenseCL-analysis

Related papers

• Improving Network Interpretability via Explanation Consistency Evaluation [56.14036428778861]
  We propose a framework that acquires more explainable activation heatmaps and simultaneously increase the model performance. Specifically, our framework introduces a new metric, i.e., explanation consistency, to reweight the training samples adaptively in model learning. Our framework then promotes the model learning by paying closer attention to those training samples with a high difference in explanations.
  arXiv  Detail & Related papers  (2024-08-08T17:20:08Z)
• Weighted Point Cloud Embedding for Multimodal Contrastive Learning Toward Optimal Similarity Metric [44.95433989446052]
  We show the benefit of our proposed method through a new understanding of the contrastive loss of CLIP. We show that our proposed similarity based on weighted point clouds consistently achieves the optimal similarity.
  arXiv  Detail & Related papers  (2024-04-30T03:15:04Z)
• Hodge-Aware Contrastive Learning [101.56637264703058]
  Simplicial complexes prove effective in modeling data with multiway dependencies. We develop a contrastive self-supervised learning approach for processing simplicial data.
  arXiv  Detail & Related papers  (2023-09-14T00:40:07Z)
• Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap [64.60460828425502]
  We propose a new guarantee on the downstream performance of contrastive learning. Our new theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations. We propose an unsupervised model selection metric ARC that aligns well with downstream accuracy.
  arXiv  Detail & Related papers  (2022-03-25T05:36:26Z)
• Understanding Contrastive Learning Requires Incorporating Inductive Biases [64.56006519908213]
  Recent attempts to theoretically explain the success of contrastive learning on downstream tasks prove guarantees depending on properties of em augmentations and the value of em contrastive loss of representations. We demonstrate that such analyses ignore em inductive biases of the function class and training algorithm, even em provably leading to vacuous guarantees in some settings.
  arXiv  Detail & Related papers  (2022-02-28T18:59:20Z)
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss [72.62029620566925]
  Recent works in self-supervised learning have advanced the state-of-the-art by relying on the contrastive learning paradigm. Our work analyzes contrastive learning without assuming conditional independence of positive pairs. We propose a loss that performs spectral decomposition on the population augmentation graph and can be succinctly written as a contrastive learning objective.
  arXiv  Detail & Related papers  (2021-06-08T07:41:02Z)
• Disentangled Contrastive Learning for Learning Robust Textual Representations [13.880693856907037]
  We introduce the concept of momentum representation consistency to align features and leverage power normalization while conforming the uniformity. Our experimental results for the NLP benchmarks demonstrate that our approach can obtain better results compared with the baselines.
  arXiv  Detail & Related papers  (2021-04-11T03:32:49Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere [32.0469500831667]
  We identify two key properties related to the contrastive loss. We prove that the contrastive loss optimize these properties, and analyze their positive effects on downstream tasks. Remarkably, directly optimizing for these two metrics leads to representations with comparable or better performance at downstream tasks than contrastive learning.
  arXiv  Detail & Related papers  (2020-05-20T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.