Contrastive Representation Learning: A Framework and Review
- URL: http://arxiv.org/abs/2010.05113v2
- Date: Tue, 27 Oct 2020 21:52:21 GMT
- Title: Contrastive Representation Learning: A Framework and Review
- Authors: Phuc H. Le-Khac, Graham Healy, Alan F. Smeaton
- Abstract summary: The origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields.
We propose a general Contrastive Representation Learning framework that simplifies and unifies many different contrastive learning methods.
Examples of how contrastive learning has been applied in computer vision, natural language processing, audio processing, and others, as well as in Reinforcement Learning are also presented.
- Score: 2.7393821783237184
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Contrastive Learning has recently received interest due to its success in
self-supervised representation learning in the computer vision domain. However,
the origins of Contrastive Learning date as far back as the 1990s and its
development has spanned across many fields and domains including Metric
Learning and natural language processing. In this paper we provide a
comprehensive literature review and we propose a general Contrastive
Representation Learning framework that simplifies and unifies many different
contrastive learning methods. We also provide a taxonomy for each of the
components of contrastive learning in order to summarise it and distinguish it
from other forms of machine learning. We then discuss the inductive biases
which are present in any contrastive learning system and we analyse our
framework under different views from various sub-fields of Machine Learning.
Examples of how contrastive learning has been applied in computer vision,
natural language processing, audio processing, and others, as well as in
Reinforcement Learning are also presented. Finally, we discuss the challenges
and some of the most promising future research directions ahead.
Related papers
- Heterogeneous Contrastive Learning for Foundation Models and Beyond [73.74745053250619]
In the era of big data and Artificial Intelligence, an emerging paradigm is to utilize contrastive self-supervised learning to model large-scale heterogeneous data.
This survey critically evaluates the current landscape of heterogeneous contrastive learning for foundation models.
arXiv Detail & Related papers (2024-03-30T02:55:49Z) - Understanding the Progression of Educational Topics via Semantic Matching [0.9246281666115259]
Education systems are dynamically changing to accommodate technological advances, industrial and societal needs, and to enhance students' learning journeys.
Curriculum specialists and educators constantly revise taught subjects across educational grades to identify gaps, introduce new learning topics, and enhance the learning outcomes.
Having nuanced data about subjects, topics, and learning outcomes structured within a dataset, empowers us to leverage data science to better understand the progression of various learning topics.
arXiv Detail & Related papers (2024-02-10T08:24:29Z) - A Message Passing Perspective on Learning Dynamics of Contrastive
Learning [60.217972614379065]
We show that if we cast a contrastive objective equivalently into the feature space, then its learning dynamics admits an interpretable form.
This perspective also establishes an intriguing connection between contrastive learning and Message Passing Graph Neural Networks (MP-GNNs)
arXiv Detail & Related papers (2023-03-08T08:27:31Z) - Learning Visual Representations via Language-Guided Sampling [25.117909306792324]
We use language similarity to sample semantically similar image pairs for contrastive learning.
Our approach diverges from image-based contrastive learning by sampling view pairs using language similarity.
We show that language-guided learning yields better features than image-based and image-text representation learning approaches.
arXiv Detail & Related papers (2023-02-23T18:59:05Z) - Localization vs. Semantics: Visual Representations in Unimodal and
Multimodal Models [57.08925810659545]
We conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models.
Our empirical observations suggest that vision-and-language models are better at label prediction tasks.
We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.
arXiv Detail & Related papers (2022-12-01T05:00:18Z) - Vision+X: A Survey on Multimodal Learning in the Light of Data [64.03266872103835]
multimodal machine learning that incorporates data from various sources has become an increasingly popular research area.
We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions.
We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels.
arXiv Detail & Related papers (2022-10-05T13:14:57Z) - Self-Supervised Speech Representation Learning: A Review [105.1545308184483]
Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains.
Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods.
This review presents approaches for self-supervised speech representation learning and their connection to other research areas.
arXiv Detail & Related papers (2022-05-21T16:52:57Z) - Semi-Supervising Learning, Transfer Learning, and Knowledge Distillation
with SimCLR [2.578242050187029]
Recent breakthroughs in the field of semi-supervised learning have achieved results that match state-of-the-art traditional supervised learning methods.
SimCLR is the current state-of-the-art semi-supervised learning framework for computer vision.
arXiv Detail & Related papers (2021-08-02T01:37:39Z) - Visual Probing: Cognitive Framework for Explaining Self-Supervised Image
Representations [12.485001250777248]
Recently introduced self-supervised methods for image representation learning provide on par or superior results to their fully supervised competitors.
Motivated by this observation, we introduce a novel visual probing framework for explaining the self-supervised models.
We show the effectiveness and applicability of those analogs in the context of explaining self-supervised representations.
arXiv Detail & Related papers (2021-06-21T12:40:31Z) - Revisiting Meta-Learning as Supervised Learning [69.2067288158133]
We aim to provide a principled, unifying framework by revisiting and strengthening the connection between meta-learning and traditional supervised learning.
By treating pairs of task-specific data sets and target models as (feature, label) samples, we can reduce many meta-learning algorithms to instances of supervised learning.
This view not only unifies meta-learning into an intuitive and practical framework but also allows us to transfer insights from supervised learning directly to improve meta-learning.
arXiv Detail & Related papers (2020-02-03T06:13:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.