Contrast All the Time: Learning Time Series Representation from Temporal Consistency

• URL: http://arxiv.org/abs/2410.15416v2
• Date: Tue, 15 Jul 2025 13:42:19 GMT
• Title: Contrast All the Time: Learning Time Series Representation from Temporal Consistency
• Authors: Abdul-Kazeem Shamba, Kerstin Bach, Gavin Taylor,
• Abstract summary: CaTT is a new approach to unsupervised contrastive learning for time series.<n>It takes advantage of dynamics between temporally similar moments more efficiently and effectively than existing methods.
• Score: 6.086030037869592
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Representation learning for time series using contrastive learning has emerged as a critical technique for improving the performance of downstream tasks. To advance this effective approach, we introduce CaTT (\textit{Contrast All The Time}), a new approach to unsupervised contrastive learning for time series, which takes advantage of dynamics between temporally similar moments more efficiently and effectively than existing methods. CaTT departs from conventional time-series contrastive approaches that rely on data augmentations or selected views. Instead, it uses the full temporal dimension by contrasting all time steps in parallel. This is made possible by a scalable NT-pair formulation, which extends the classic N-pair loss across both batch and temporal dimensions, making the learning process end-to-end and more efficient. CaTT learns directly from the natural structure of temporal data, using repeated or adjacent time steps as implicit supervision, without the need for pair selection heuristics. We demonstrate that this approach produces superior embeddings which allow better performance in downstream tasks. Additionally, training is faster than other contrastive learning approaches, making it suitable for large-scale and real-world time series applications. The source code is publicly available at \href{https://github.com/sfi-norwai/CaTT}{https://github.com/sfi-norwai/CaTT}.

Related papers

• Frequency-Masked Embedding Inference: A Non-Contrastive Approach for Time Series Representation Learning [0.38366697175402226]
  This paper introduces Frequency-masked Embedding Inference (FEI), a novel non-contrastive method that completely eliminates the need for positive and negative samples. FEI significantly outperforms existing contrastive-based methods in terms of generalization. This study provides new insights into self-supervised representation learning for time series.
  arXiv  Detail & Related papers  (2024-12-30T08:12:17Z)
• Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making [66.27188304203217]
  Temporal distances lie at the heart of many algorithms for planning, control, and reinforcement learning.<n>Prior attempts to define such temporal distances in settings have been stymied by an important limitation.<n>We show how successor features learned by contrastive learning form a temporal distance that does satisfy the triangle inequality.
  arXiv  Detail & Related papers  (2024-06-24T19:36:45Z)
• Capturing Temporal Components for Time Series Classification [5.70772577110828]
  This work introduces a textitcompositional representation learning approach trained on statistically coherent components extracted from sequential data. Based on a multi-scale change space, an unsupervised approach is proposed to segment the sequential data into chunks with similar statistical properties. A sequence-based encoder model is trained in a multi-task setting to learn compositional representations from these temporal components for time series classification.
  arXiv  Detail & Related papers  (2024-06-20T16:15:21Z)
• Time Series Representation Learning with Supervised Contrastive Temporal Transformer [8.223940676615857]
  We develop a simple, yet novel fusion model, called: textbfSupervised textbfCOntrastive textbfTemporal textbfTransformer (SCOTT) We first investigate suitable augmentation methods for various types of time series data to assist with learning change-invariant representations.
  arXiv  Detail & Related papers  (2024-03-16T03:37:19Z)
• TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling [67.02157180089573]
  Time series pre-training has recently garnered wide attention for its potential to reduce labeling expenses and benefit various downstream tasks. This paper proposes TimeSiam as a simple but effective self-supervised pre-training framework for Time series based on Siamese networks.
  arXiv  Detail & Related papers  (2024-02-04T13:10:51Z)
• Distillation Enhanced Time Series Forecasting Network with Momentum Contrastive Learning [7.4106801792345705]
  We propose DE-TSMCL, an innovative distillation enhanced framework for long sequence time series forecasting. Specifically, we design a learnable data augmentation mechanism which adaptively learns whether to mask a timestamp. Then, we propose a contrastive learning task with momentum update to explore inter-sample and intra-temporal correlations of time series. By developing model loss from multiple tasks, we can learn effective representations for downstream forecasting task.
  arXiv  Detail & Related papers  (2024-01-31T12:52:10Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Soft Contrastive Learning for Time Series [5.752266579415516]
  We propose SoftCLT, a simple yet effective soft contrastive learning strategy for time series. Specifically, we define soft assignments for 1) instance-wise contrastive loss by the distance between time series on the data space, and 2) temporal contrastive loss by the difference of timestamps. In experiments, we demonstrate that SoftCLT consistently improves the performance in various downstream tasks including classification, semi-supervised learning, transfer learning, and anomaly detection.
  arXiv  Detail & Related papers  (2023-12-27T06:15:00Z)
• Contrastive Difference Predictive Coding [79.74052624853303]
  We introduce a temporal difference version of contrastive predictive coding that stitches together pieces of different time series data to decrease the amount of data required to learn predictions of future events. We apply this representation learning method to derive an off-policy algorithm for goal-conditioned RL.
  arXiv  Detail & Related papers  (2023-10-31T03:16:32Z)
• Multi-Task Self-Supervised Time-Series Representation Learning [3.31490164885582]
  Time-series representation learning can extract representations from data with temporal dynamics and sparse labels. We propose a new time-series representation learning method by combining the advantages of self-supervised tasks. We evaluate the proposed framework on three downstream tasks: time-series classification, forecasting, and anomaly detection.
  arXiv  Detail & Related papers  (2023-03-02T07:44:06Z)
• MHCCL: Masked Hierarchical Cluster-Wise Contrastive Learning for Multivariate Time Series [20.008535430484475]
  Masked Hierarchical Cluster-wise Contrastive Learning model is presented. It exploits semantic information obtained from the hierarchical structure consisting of multiple latent partitions for time series. It is shown to be superior to state-of-the-art approaches for unsupervised time series representation learning.
  arXiv  Detail & Related papers  (2022-12-02T12:42:53Z)
• Learning Sequence Representations by Non-local Recurrent Neural Memory [61.65105481899744]
  We propose a Non-local Recurrent Neural Memory (NRNM) for supervised sequence representation learning. Our model is able to capture long-range dependencies and latent high-level features can be distilled by our model. Our model compares favorably against other state-of-the-art methods specifically designed for each of these sequence applications.
  arXiv  Detail & Related papers  (2022-07-20T07:26:15Z)
• Towards Similarity-Aware Time-Series Classification [51.2400839966489]
  We study time-series classification (TSC), a fundamental task of time-series data mining. We propose Similarity-Aware Time-Series Classification (SimTSC), a framework that models similarity information with graph neural networks (GNNs)
  arXiv  Detail & Related papers  (2022-01-05T02:14:57Z)
• Time-Series Representation Learning via Temporal and Contextual Contrasting [14.688033556422337]
  We propose an unsupervised Time-Series representation learning framework via Temporal and Contextual Contrasting (TS-TCC) First, the raw time-series data are transformed into two different yet correlated views by using weak and strong augmentations. Second, we propose a novel temporal contrasting module to learn robust temporal representations by designing a tough cross-view prediction task. Third, to further learn discriminative representations, we propose a contextual contrasting module built upon the contexts from the temporal contrasting module.
  arXiv  Detail & Related papers  (2021-06-26T23:56:31Z)
• You Never Cluster Alone [150.94921340034688]
  We extend the mainstream contrastive learning paradigm to a cluster-level scheme, where all the data subjected to the same cluster contribute to a unified representation. We define a set of categorical variables as clustering assignment confidence, which links the instance-level learning track with the cluster-level one. By reparametrizing the assignment variables, TCC is trained end-to-end, requiring no alternating steps.
  arXiv  Detail & Related papers  (2021-06-03T14:59:59Z)
• Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding [8.45908939323268]
  We propose a self-supervised framework for learning generalizable representations for non-stationary time series. Our motivation stems from the medical field, where the ability to model the dynamic nature of time series data is especially valuable.
  arXiv  Detail & Related papers  (2021-06-01T19:53:24Z)
• Interpretable Time-series Representation Learning With Multi-Level Disentanglement [56.38489708031278]
  Disentangle Time Series (DTS) is a novel disentanglement enhancement framework for sequential data. DTS generates hierarchical semantic concepts as the interpretable and disentangled representation of time-series. DTS achieves superior performance in downstream applications, with high interpretability of semantic concepts.
  arXiv  Detail & Related papers  (2021-05-17T22:02:24Z)
• Contrastive learning of strong-mixing continuous-time stochastic processes [53.82893653745542]
  Contrastive learning is a family of self-supervised methods where a model is trained to solve a classification task constructed from unlabeled data. We show that a properly constructed contrastive learning task can be used to estimate the transition kernel for small-to-mid-range intervals in the diffusion case.
  arXiv  Detail & Related papers  (2021-03-03T23:06:47Z)
• Temporal Contrastive Graph Learning for Video Action Recognition and Retrieval [83.56444443849679]
  This work takes advantage of the temporal dependencies within videos and proposes a novel self-supervised method named Temporal Contrastive Graph Learning (TCGL) Our TCGL roots in a hybrid graph contrastive learning strategy to jointly regard the inter-snippet and intra-snippet temporal dependencies as self-supervision signals for temporal representation learning. Experimental results demonstrate the superiority of our TCGL over the state-of-the-art methods on large-scale action recognition and video retrieval benchmarks.
  arXiv  Detail & Related papers  (2021-01-04T08:11:39Z)
• ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning [52.831894583501395]
  Continual learning assumes the incoming data are fully labeled, which might not be applicable in real applications. We propose deep Online Replay with Discriminator Consistency (ORDisCo) to interdependently learn a classifier with a conditional generative adversarial network (GAN) We show ORDisCo achieves significant performance improvement on various semi-supervised learning benchmark datasets for SSCL.
  arXiv  Detail & Related papers  (2021-01-02T09:04:14Z)
• Contrastive Learning with Adversarial Examples [79.39156814887133]
  Contrastive learning (CL) is a popular technique for self-supervised learning (SSL) of visual representations. This paper introduces a new family of adversarial examples for constrastive learning and using these examples to define a new adversarial training algorithm for SSL, denoted as CLAE.
  arXiv  Detail & Related papers  (2020-10-22T20:45:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.