Contrastive Learning with Consistent Representations
- URL: http://arxiv.org/abs/2302.01541v1
- Date: Fri, 3 Feb 2023 04:34:00 GMT
- Title: Contrastive Learning with Consistent Representations
- Authors: Zihu Wang, Yu Wang, Hanbin Hu, Peng Li
- Abstract summary: This paper proposes Contrastive Learning with Consistent Representations (CoCor)
CoCor is a new consistency measure, DA consistency, which dictates the mapping of augmented input data to the representation space.
The proposed techniques give rise to a semi-supervised learning framework based on bi-level optimization, achieving new state-of-the-art results for image recognition.
- Score: 8.274769259790926
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Contrastive learning demonstrates great promise for representation learning.
Data augmentations play a critical role in contrastive learning by providing
informative views of the data without needing the labels. However, the
performance of the existing works heavily relies on the quality of the employed
data augmentation (DA) functions, which are typically hand picked from a
restricted set of choices. While exploiting a diverse set of data augmentations
is appealing, the intricacies of DAs and representation learning may lead to
performance degradation. To address this challenge and allow for a systemic use
of large numbers of data augmentations, this paper proposes Contrastive
Learning with Consistent Representations (CoCor). At the core of CoCor is a new
consistency measure, DA consistency, which dictates the mapping of augmented
input data to the representation space such that these instances are mapped to
optimal locations in a way consistent to the intensity of the DA applied.
Furthermore, a data-driven approach is proposed to learn the optimal mapping
locations as a function of DA while maintaining a desired monotonic property
with respect to DA intensity. The proposed techniques give rise to a
semi-supervised learning framework based on bi-level optimization, achieving
new state-of-the-art results for image recognition.
Related papers
- Data Augmentation for Traffic Classification [54.92823760790628]
Data Augmentation (DA) is a technique widely adopted in Computer Vision (CV) and Natural Language Processing (NLP) tasks.
DA has struggled to gain traction in networking contexts, particularly in Traffic Classification (TC) tasks.
arXiv Detail & Related papers (2024-01-19T15:25:09Z) - Time Series Contrastive Learning with Information-Aware Augmentations [57.45139904366001]
A key component of contrastive learning is to select appropriate augmentations imposing some priors to construct feasible positive samples.
How to find the desired augmentations of time series data that are meaningful for given contrastive learning tasks and datasets remains an open question.
We propose a new contrastive learning approach with information-aware augmentations, InfoTS, that adaptively selects optimal augmentations for time series representation learning.
arXiv Detail & Related papers (2023-03-21T15:02:50Z) - Investigating Enhancements to Contrastive Predictive Coding for Human
Activity Recognition [7.086647707011785]
Contrastive Predictive Coding (CPC) is a technique that learns effective representations by leveraging properties of time-series data.
In this work, we propose enhancements to CPC, by systematically investigating the architecture, the aggregator network, and the future timestep prediction.
Our method shows substantial improvements on four of six target datasets, demonstrating its ability to empower a wide range of application scenarios.
arXiv Detail & Related papers (2022-11-11T12:54:58Z) - Joint Data and Feature Augmentation for Self-Supervised Representation
Learning on Point Clouds [4.723757543677507]
We propose a fusion contrastive learning framework to combine data augmentations in Euclidean space and feature augmentations in feature space.
We conduct extensive object classification experiments and object part segmentation experiments to validate the transferability of the proposed framework.
Experimental results demonstrate that the proposed framework is effective to learn the point cloud representation in a self-supervised manner.
arXiv Detail & Related papers (2022-11-02T14:58:03Z) - Cluster-level pseudo-labelling for source-free cross-domain facial
expression recognition [94.56304526014875]
We propose the first Source-Free Unsupervised Domain Adaptation (SFUDA) method for Facial Expression Recognition (FER)
Our method exploits self-supervised pretraining to learn good feature representations from the target data.
We validate the effectiveness of our method in four adaptation setups, proving that it consistently outperforms existing SFUDA methods when applied to FER.
arXiv Detail & Related papers (2022-10-11T08:24:50Z) - Con$^{2}$DA: Simplifying Semi-supervised Domain Adaptation by Learning
Consistent and Contrastive Feature Representations [1.2891210250935146]
Con$2$DA is a framework that extends recent advances in semi-supervised learning to the semi-supervised domain adaptation problem.
Our framework generates pairs of associated samples by performing data transformations to a given input.
We use different loss functions to enforce consistency between the feature representations of associated data pairs of samples.
arXiv Detail & Related papers (2022-04-04T15:05:45Z) - Robust Representation Learning via Perceptual Similarity Metrics [18.842322467828502]
Contrastive Input Morphing (CIM) is a representation learning framework that learns input-space transformations of the data.
We show that CIM is complementary to other mutual information-based representation learning techniques.
arXiv Detail & Related papers (2021-06-11T21:45:44Z) - Heterogeneous Contrastive Learning: Encoding Spatial Information for
Compact Visual Representations [183.03278932562438]
This paper presents an effective approach that adds spatial information to the encoding stage to alleviate the learning inconsistency between the contrastive objective and strong data augmentation operations.
We show that our approach achieves higher efficiency in visual representations and thus delivers a key message to inspire the future research of self-supervised visual representation learning.
arXiv Detail & Related papers (2020-11-19T16:26:25Z) - Graph Contrastive Learning with Adaptive Augmentation [23.37786673825192]
We propose a novel graph contrastive representation learning method with adaptive augmentation.
Specifically, we design augmentation schemes based on node centrality measures to highlight important connective structures.
Our proposed method consistently outperforms existing state-of-the-art baselines and even surpasses some supervised counterparts.
arXiv Detail & Related papers (2020-10-27T15:12:21Z) - CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for
Natural Language Understanding [67.61357003974153]
We propose a novel data augmentation framework dubbed CoDA.
CoDA synthesizes diverse and informative augmented examples by integrating multiple transformations organically.
A contrastive regularization objective is introduced to capture the global relationship among all the data samples.
arXiv Detail & Related papers (2020-10-16T23:57:03Z) - Spectrum-Guided Adversarial Disparity Learning [52.293230153385124]
We propose a novel end-to-end knowledge directed adversarial learning framework.
It portrays the class-conditioned intraclass disparity using two competitive encoding distributions and learns the purified latent codes by denoising learned disparity.
The experiments on four HAR benchmark datasets demonstrate the robustness and generalization of our proposed methods over a set of state-of-the-art.
arXiv Detail & Related papers (2020-07-14T05:46:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.