Spectral Temporal Contrastive Learning
- URL: http://arxiv.org/abs/2312.00966v2
- Date: Thu, 7 Dec 2023 20:57:07 GMT
- Title: Spectral Temporal Contrastive Learning
- Authors: Sacha Morin, Somjit Nath, Samira Ebrahimi Kahou and Guy Wolf
- Abstract summary: This work is concerned with the temporal contrastive learning setting where the sequential structure of the data is used instead to define positive pairs.
We discuss a population loss based on a state graph derived from a time-homogeneous reversible Markov chain with uniform stationary distribution.
- Score: 16.071429029573682
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Learning useful data representations without requiring labels is a
cornerstone of modern deep learning. Self-supervised learning methods,
particularly contrastive learning (CL), have proven successful by leveraging
data augmentations to define positive pairs. This success has prompted a number
of theoretical studies to better understand CL and investigate theoretical
bounds for downstream linear probing tasks. This work is concerned with the
temporal contrastive learning (TCL) setting where the sequential structure of
the data is used instead to define positive pairs, which is more commonly used
in RL and robotics contexts. In this paper, we adapt recent work on Spectral CL
to formulate Spectral Temporal Contrastive Learning (STCL). We discuss a
population loss based on a state graph derived from a time-homogeneous
reversible Markov chain with uniform stationary distribution. The STCL loss
enables to connect the linear probing performance to the spectral properties of
the graph, and can be estimated by considering previously observed data
sequences as an ensemble of MCMC chains.
Related papers
- Learning Time-aware Graph Structures for Spatially Correlated Time
Series Forecasting [30.93275270960829]
We propose Time-aware Graph Structure Learning (TagSL), which extracts time-aware correlations among time series.
We also present a Graph Convolution-based Gated Recurrent Unit (GCGRU), that jointly captures spatial and temporal dependencies.
Finally, we introduce a unified framework named Time-aware Graph Convolutional Recurrent Network (TGCRN), combining TagSL, GCGRU in an encoder-decoder architecture for multi-step-temporal forecasting.
arXiv Detail & Related papers (2023-12-27T04:23:43Z) - STS-CCL: Spatial-Temporal Synchronous Contextual Contrastive Learning
for Urban Traffic Forecasting [4.947443433688782]
This work employs the advanced contrastive learning and proposes a novel Spatial-Temporalous Contextual Contrastive Learning (STS-CCL) model.
Experiments and evaluations demonstrate that building a predictor upon STS-CCL contrastive learning model gains superior performance than existing traffic forecasting benchmarks.
arXiv Detail & Related papers (2023-07-05T03:47:28Z) - HomoGCL: Rethinking Homophily in Graph Contrastive Learning [64.85392028383164]
HomoGCL is a model-agnostic framework to expand the positive set using neighbor nodes with neighbor-specific significances.
We show that HomoGCL yields multiple state-of-the-art results across six public datasets.
arXiv Detail & Related papers (2023-06-16T04:06:52Z) - Understanding Multimodal Contrastive Learning and Incorporating Unpaired
Data [19.72282903349282]
We show a general class of nonlinear loss functions for multimodal contrastive learning (MMCL)
We quantitatively show that the feature learning ability of MMCL can be better than that of unimodal contrastive learning applied to each modality.
When we have access to additional unpaired data, we propose a new MMCL loss that incorporates additional unpaired datasets.
arXiv Detail & Related papers (2023-02-13T10:11:05Z) - Single-Pass Contrastive Learning Can Work for Both Homophilic and
Heterophilic Graph [60.28340453547902]
Graph contrastive learning (GCL) techniques typically require two forward passes for a single instance to construct the contrastive loss.
Existing GCL approaches fail to provide strong performance guarantees.
We implement the Single-Pass Graph Contrastive Learning method (SP-GCL)
Empirically, the features learned by the SP-GCL can match or outperform existing strong baselines with significantly less computational overhead.
arXiv Detail & Related papers (2022-11-20T07:18:56Z) - Spectral Regularization Allows Data-frugal Learning over Combinatorial
Spaces [13.36217184117654]
We show that regularizing the spectral representation of machine learning models improves their generalization power when labeled data is scarce.
Running gradient descent on the regularized loss results in a better generalization performance compared to baseline algorithms in several data-scarce real-world problems.
arXiv Detail & Related papers (2022-10-05T23:31:54Z) - The CLEAR Benchmark: Continual LEArning on Real-World Imagery [77.98377088698984]
Continual learning (CL) is widely regarded as crucial challenge for lifelong AI.
We introduce CLEAR, the first continual image classification benchmark dataset with a natural temporal evolution of visual concepts.
We find that a simple unsupervised pre-training step can already boost state-of-the-art CL algorithms.
arXiv Detail & Related papers (2022-01-17T09:09:09Z) - Provable Guarantees for Self-Supervised Deep Learning with Spectral
Contrastive Loss [72.62029620566925]
Recent works in self-supervised learning have advanced the state-of-the-art by relying on the contrastive learning paradigm.
Our work analyzes contrastive learning without assuming conditional independence of positive pairs.
We propose a loss that performs spectral decomposition on the population augmentation graph and can be succinctly written as a contrastive learning objective.
arXiv Detail & Related papers (2021-06-08T07:41:02Z) - ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for
Semi-supervised Continual Learning [52.831894583501395]
Continual learning assumes the incoming data are fully labeled, which might not be applicable in real applications.
We propose deep Online Replay with Discriminator Consistency (ORDisCo) to interdependently learn a classifier with a conditional generative adversarial network (GAN)
We show ORDisCo achieves significant performance improvement on various semi-supervised learning benchmark datasets for SSCL.
arXiv Detail & Related papers (2021-01-02T09:04:14Z) - New advances in enumerative biclustering algorithms with online
partitioning [80.22629846165306]
This paper further extends RIn-Close_CVC, a biclustering algorithm capable of performing an efficient, complete, correct and non-redundant enumeration of maximal biclusters with constant values on columns in numerical datasets.
The improved algorithm is called RIn-Close_CVC3, keeps those attractive properties of RIn-Close_CVC, and is characterized by: a drastic reduction in memory usage; a consistent gain in runtime.
arXiv Detail & Related papers (2020-03-07T14:54:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.