Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations

• URL: http://arxiv.org/abs/2204.02683v1
• Date: Wed, 6 Apr 2022 09:10:23 GMT
• Title: Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations
• Authors: Jeff Z. HaoChen, Colin Wei, Ananya Kumar, Tengyu Ma
• Abstract summary: Contrastive learning is a highly effective method which uses unlabeled data to produce representations which are linearly separable for downstream classification tasks. Recent works have shown that contrastive representations are not only useful when data come from a single domain, but are also effective for transferring across domains.
• Score: 50.33975968859988
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastive learning is a highly effective method which uses unlabeled data to produce representations which are linearly separable for downstream classification tasks. Recent works have shown that contrastive representations are not only useful when data come from a single domain, but are also effective for transferring across domains. Concretely, when contrastive representations are trained on data from two domains (a source and target) and a linear classification head is trained to predict labels using only the labeled source data, the resulting classifier also exhibits good transfer to the target domain. In this work, we analyze this linear transferability phenomenon, building upon the framework proposed by HaoChen et al (2021) which relates contrastive learning to spectral clustering of a positive-pair graph on the data. We prove that contrastive representations capture relationships between subpopulations in the positive-pair graph: linear transferability can occur when data from the same class in different domains (e.g., photo dogs and cartoon dogs) are connected in the graph. Our analysis allows the source and target classes to have unbounded density ratios and be mapped to distant representations. Our proof is also built upon technical improvements over the main results of HaoChen et al (2021), which may be of independent interest.

Related papers

• kNN Classification of Malware Data Dependency Graph Features [0.0]
  This study obtains accurate classification from the use of features tied to structure and semantics. By training an accurate model using labeled data, this feature representation of semantics is shown to be correlated with ground truth labels. Our results provide evidence that data dependency graphs accurately capture both semantic and structural information for increased explainability in classification results.
  arXiv  Detail & Related papers  (2024-06-04T16:39:02Z)
• Domain Adaptive Graph Classification [0.0]
  We introduce the Dual Adversarial Graph Representation Learning (DAGRL), which explore the graph topology from dual branches and mitigate domain discrepancies via dual adversarial learning. Our approach incorporates adaptive perturbations into the dual branches, which align the source and target distribution to address domain discrepancies.
  arXiv  Detail & Related papers  (2023-12-21T02:37:56Z)
• The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning [32.15608637930748]
  We show that there exists a trade-off between the two desiderata so that one may not be able to achieve both simultaneously. We provide analysis using a theoretical data model and show that, while more diverse pre-training data result in more diverse features for different tasks, it puts less emphasis on task-specific features.
  arXiv  Detail & Related papers  (2023-02-28T22:14:33Z)
• You Only Transfer What You Share: Intersection-Induced Graph Transfer Learning for Link Prediction [79.15394378571132]
  We investigate a previously overlooked phenomenon: in many cases, a densely connected, complementary graph can be found for the original graph. The denser graph may share nodes with the original graph, which offers a natural bridge for transferring selective, meaningful knowledge. We identify this setting as Graph Intersection-induced Transfer Learning (GITL), which is motivated by practical applications in e-commerce or academic co-authorship predictions.
  arXiv  Detail & Related papers  (2023-02-27T22:56:06Z)
• Metric Distribution to Vector: Constructing Data Representation via Broad-Scale Discrepancies [15.40538348604094]
  We present a novel embedding strategy named $mathbfMetricDistribution2vec$ to extract distribution characteristics into the vectorial representation for each data. We demonstrate the application and effectiveness of our representation method in the supervised prediction tasks on extensive real-world structural graph datasets.
  arXiv  Detail & Related papers  (2022-10-02T03:18:30Z)
• Similarity-aware Positive Instance Sampling for Graph Contrastive Pre-training [82.68805025636165]
  We propose to select positive graph instances directly from existing graphs in the training set. Our selection is based on certain domain-specific pair-wise similarity measurements. Besides, we develop an adaptive node-level pre-training method to dynamically mask nodes to distribute them evenly in the graph.
  arXiv  Detail & Related papers  (2022-06-23T20:12:51Z)
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss [72.62029620566925]
  Recent works in self-supervised learning have advanced the state-of-the-art by relying on the contrastive learning paradigm. Our work analyzes contrastive learning without assuming conditional independence of positive pairs. We propose a loss that performs spectral decomposition on the population augmentation graph and can be succinctly written as a contrastive learning objective.
  arXiv  Detail & Related papers  (2021-06-08T07:41:02Z)
• TraND: Transferable Neighborhood Discovery for Unsupervised Cross-domain Gait Recognition [77.77786072373942]
  This paper proposes a Transferable Neighborhood Discovery (TraND) framework to bridge the domain gap for unsupervised cross-domain gait recognition. We design an end-to-end trainable approach to automatically discover the confident neighborhoods of unlabeled samples in the latent space. Our method achieves state-of-the-art results on two public datasets, i.e., CASIA-B and OU-LP.
  arXiv  Detail & Related papers  (2021-02-09T03:07:07Z)
• i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning [117.63815437385321]
  We propose i-Mix, a simple yet effective domain-agnostic regularization strategy for improving contrastive representation learning. In experiments, we demonstrate that i-Mix consistently improves the quality of learned representations across domains.
  arXiv  Detail & Related papers  (2020-10-17T23:32:26Z)
• Geometric graphs from data to aid classification tasks with graph convolutional networks [0.0]
  We show that, even if additional relational information is not available in the data set, one can improve classification by constructing geometric graphs from the features themselves. The improvement in classification accuracy is maximized by graphs that capture sample similarity with relatively low edge density.
  arXiv  Detail & Related papers  (2020-05-08T15:00:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.