TimeDRL: Disentangled Representation Learning for Multivariate Time-Series

• URL: http://arxiv.org/abs/2312.04142v3
• Date: Wed, 17 Jul 2024 15:19:17 GMT
• Title: TimeDRL: Disentangled Representation Learning for Multivariate Time-Series
• Authors: Ching Chang, Chiao-Tung Chan, Wei-Yao Wang, Wen-Chih Peng, Tien-Fu Chen,
• Abstract summary: TimeDRL is a generic time-series representation learning framework with disentangled dual-level embeddings. TimeDRL consistently surpasses existing representation learning approaches, achieving an average improvement of 58.02% in MSE and classification by 1.48% in accuracy.
• Score: 10.99576829280084
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multivariate time-series data in numerous real-world applications (e.g., healthcare and industry) are informative but challenging due to the lack of labels and high dimensionality. Recent studies in self-supervised learning have shown their potential in learning rich representations without relying on labels, yet they fall short in learning disentangled embeddings and addressing issues of inductive bias (e.g., transformation-invariance). To tackle these challenges, we propose TimeDRL, a generic multivariate time-series representation learning framework with disentangled dual-level embeddings. TimeDRL is characterized by three novel features: (i) disentangled derivation of timestamp-level and instance-level embeddings from patched time-series data using a [CLS] token strategy; (ii) utilization of timestamp-predictive and instance-contrastive tasks for disentangled representation learning, with the former optimizing timestamp-level embeddings with predictive loss, and the latter optimizing instance-level embeddings with contrastive loss; and (iii) avoidance of augmentation methods to eliminate inductive biases, such as transformation-invariance from cropping and masking. Comprehensive experiments on 6 time-series forecasting datasets and 5 time-series classification datasets have shown that TimeDRL consistently surpasses existing representation learning approaches, achieving an average improvement of forecasting by 58.02% in MSE and classification by 1.48% in accuracy. Furthermore, extensive ablation studies confirmed the relative contribution of each component in TimeDRL's architecture, and semi-supervised learning evaluations demonstrated its effectiveness in real-world scenarios, even with limited labeled data. The code is available at https://github.com/blacksnail789521/TimeDRL.

Related papers

• Self-Supervised Learning of Disentangled Representations for Multivariate Time-Series [10.99576829280084]
  TimeDRL is a framework for multivariate time-series representation learning with dual-level disentangled embeddings. TimeDRL features: (i) timestamp-level and instance-level embeddings using a [] token strategy; (ii) timestamp-predictive and instance-contrastive tasks for representation learning; and (iii) avoidance of augmentation methods to eliminate inductive biases. Experiments on forecasting and classification datasets show TimeDRL outperforms existing methods, with further validation in semi-supervised settings with limited labeled data.
  arXiv  Detail & Related papers  (2024-10-16T14:24:44Z)
• Graph Spatiotemporal Process for Multivariate Time Series Anomaly Detection with Missing Values [67.76168547245237]
  We introduce a novel framework called GST-Pro, which utilizes a graphtemporal process and anomaly scorer to detect anomalies. Our experimental results show that the GST-Pro method can effectively detect anomalies in time series data and outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-01-11T10:10:16Z)
• T-Rep: Representation Learning for Time Series using Time-Embeddings [5.885238773559017]
  We propose T-Rep, a self-supervised method to learn time series representations at a timestep granularity. T-Rep learns vector embeddings of time alongside its feature extractor, to extract temporal features. We evaluate T-Rep on downstream classification, forecasting, and anomaly detection tasks.
  arXiv  Detail & Related papers  (2023-10-06T15:45:28Z)
• DIVERSIFY: A General Framework for Time Series Out-of-distribution Detection and Generalization [58.704753031608625]
  Time series is one of the most challenging modalities in machine learning research. OOD detection and generalization on time series tend to suffer due to its non-stationary property. We propose DIVERSIFY, a framework for OOD detection and generalization on dynamic distributions of time series.
  arXiv  Detail & Related papers  (2023-08-04T12:27:11Z)
• Time Series Contrastive Learning with Information-Aware Augmentations [57.45139904366001]
  A key component of contrastive learning is to select appropriate augmentations imposing some priors to construct feasible positive samples. How to find the desired augmentations of time series data that are meaningful for given contrastive learning tasks and datasets remains an open question. We propose a new contrastive learning approach with information-aware augmentations, InfoTS, that adaptively selects optimal augmentations for time series representation learning.
  arXiv  Detail & Related papers  (2023-03-21T15:02:50Z)
• Time Associated Meta Learning for Clinical Prediction [78.99422473394029]
  We propose a novel time associated meta learning (TAML) method to make effective predictions at multiple future time points. To address the sparsity problem after task splitting, TAML employs a temporal information sharing strategy to augment the number of positive samples. We demonstrate the effectiveness of TAML on multiple clinical datasets, where it consistently outperforms a range of strong baselines.
  arXiv  Detail & Related papers  (2023-03-05T03:54:54Z)
• Multi-Task Self-Supervised Time-Series Representation Learning [3.31490164885582]
  Time-series representation learning can extract representations from data with temporal dynamics and sparse labels. We propose a new time-series representation learning method by combining the advantages of self-supervised tasks. We evaluate the proposed framework on three downstream tasks: time-series classification, forecasting, and anomaly detection.
  arXiv  Detail & Related papers  (2023-03-02T07:44:06Z)
• Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time [69.77704012415845]
  Temporal shifts can considerably degrade performance of machine learning models deployed in the real world. We benchmark 13 prior approaches, including methods in domain generalization, continual learning, self-supervised learning, and ensemble learning. Under both evaluation strategies, we observe an average performance drop of 20% from in-distribution to out-of-distribution data.
  arXiv  Detail & Related papers  (2022-11-25T17:07:53Z)
• Contrastive predictive coding for Anomaly Detection in Multi-variate Time Series Data [6.463941665276371]
  We propose a Time-series Representational Learning through Contrastive Predictive Coding (TRL-CPC) towards anomaly detection in MVTS data. First, we jointly optimize an encoder, an auto-regressor and a non-linear transformation function to effectively learn the representations of the MVTS data sets.
  arXiv  Detail & Related papers  (2022-02-08T04:25:29Z)
• Self-supervised Transformer for Multivariate Clinical Time-Series with Missing Values [7.9405251142099464]
  We present STraTS (Self-supervised Transformer for TimeSeries) model. It treats time-series as a set of observation triplets instead of using the traditional dense matrix representation. It shows better prediction performance than state-of-theart methods for mortality prediction, especially when labeled data is limited.
  arXiv  Detail & Related papers  (2021-07-29T19:39:39Z)
• Interpretable Time-series Representation Learning With Multi-Level Disentanglement [56.38489708031278]
  Disentangle Time Series (DTS) is a novel disentanglement enhancement framework for sequential data. DTS generates hierarchical semantic concepts as the interpretable and disentangled representation of time-series. DTS achieves superior performance in downstream applications, with high interpretability of semantic concepts.
  arXiv  Detail & Related papers  (2021-05-17T22:02:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.