An Augmentation Overlap Theory of Contrastive Learning

• URL: http://arxiv.org/abs/2511.03114v1
• Date: Wed, 05 Nov 2025 01:45:48 GMT
• Title: An Augmentation Overlap Theory of Contrastive Learning
• Authors: Qi Zhang, Yifei Wang, Yisen Wang,
• Abstract summary: Self-supervised contrastive learning has achieved great success on various tasks.<n>In this paper, we provide the tightest bounds based on the widely adopted assumption of conditional independence.<n>We also relax the conditional independence assumption to a more practical assumption of augmentation overlap.
• Score: 39.44413331664107
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, self-supervised contrastive learning has achieved great success on various tasks. However, its underlying working mechanism is yet unclear. In this paper, we first provide the tightest bounds based on the widely adopted assumption of conditional independence. Further, we relax the conditional independence assumption to a more practical assumption of augmentation overlap and derive the asymptotically closed bounds for the downstream performance. Our proposed augmentation overlap theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations, thus simply aligning the positive samples (augmented views of the same sample) could make contrastive learning cluster intra-class samples together. Moreover, from the newly derived augmentation overlap perspective, we develop an unsupervised metric for the representation evaluation of contrastive learning, which aligns well with the downstream performance almost without relying on additional modules. Code is available at https://github.com/PKU-ML/GARC.

Related papers

• BECLR: Batch Enhanced Contrastive Few-Shot Learning [1.450405446885067]
  Unsupervised few-shot learning aspires to bridge this gap by discarding the reliance on annotations at training time. We propose a novel Dynamic Clustered mEmory (DyCE) module to promote a highly separable latent representation space. We then tackle the, somehow overlooked yet critical, issue of sample bias at the few-shot inference stage.
  arXiv  Detail & Related papers  (2024-02-04T10:52:43Z)
• Unilaterally Aggregated Contrastive Learning with Hierarchical Augmentation for Anomaly Detection [64.50126371767476]
  We propose Unilaterally Aggregated Contrastive Learning with Hierarchical Augmentation (UniCon-HA) We explicitly encourage the concentration of inliers and the dispersion of virtual outliers via supervised and unsupervised contrastive losses. Our method is evaluated under three AD settings including unlabeled one-class, unlabeled multi-class, and labeled multi-class.
  arXiv  Detail & Related papers  (2023-08-20T04:01:50Z)
• Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap [64.60460828425502]
  We propose a new guarantee on the downstream performance of contrastive learning. Our new theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations. We propose an unsupervised model selection metric ARC that aligns well with downstream accuracy.
  arXiv  Detail & Related papers  (2022-03-25T05:36:26Z)
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss [72.62029620566925]
  Recent works in self-supervised learning have advanced the state-of-the-art by relying on the contrastive learning paradigm. Our work analyzes contrastive learning without assuming conditional independence of positive pairs. We propose a loss that performs spectral decomposition on the population augmentation graph and can be succinctly written as a contrastive learning objective.
  arXiv  Detail & Related papers  (2021-06-08T07:41:02Z)
• Incremental False Negative Detection for Contrastive Learning [95.68120675114878]
  We introduce a novel incremental false negative detection for self-supervised contrastive learning. During contrastive learning, we discuss two strategies to explicitly remove the detected false negatives. Our proposed method outperforms other self-supervised contrastive learning frameworks on multiple benchmarks within a limited compute.
  arXiv  Detail & Related papers  (2021-06-07T15:29:14Z)
• Doubly Contrastive Deep Clustering [135.7001508427597]
  We present a novel Doubly Contrastive Deep Clustering (DCDC) framework, which constructs contrastive loss over both sample and class views. Specifically, for the sample view, we set the class distribution of the original sample and its augmented version as positive sample pairs. For the class view, we build the positive and negative pairs from the sample distribution of the class. In this way, two contrastive losses successfully constrain the clustering results of mini-batch samples in both sample and class level.
  arXiv  Detail & Related papers  (2021-03-09T15:15:32Z)
• Beyond Single Instance Multi-view Unsupervised Representation Learning [21.449132256091662]
  We impose more accurate instance discrimination capability by measuring the joint similarity between two randomly sampled instances. We believe that learning joint similarity helps to improve the performance when encoded features are distributed more evenly in the latent space.
  arXiv  Detail & Related papers  (2020-11-26T15:43:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.