TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling

• URL: http://arxiv.org/abs/2402.02475v2
• Date: Fri, 7 Jun 2024 07:46:26 GMT
• Title: TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling
• Authors: Jiaxiang Dong, Haixu Wu, Yuxuan Wang, Yunzhong Qiu, Li Zhang, Jianmin Wang, Mingsheng Long,
• Abstract summary: Time series pre-training has recently garnered wide attention for its potential to reduce labeling expenses and benefit various downstream tasks. This paper proposes TimeSiam as a simple but effective self-supervised pre-training framework for Time series based on Siamese networks.
• Score: 67.02157180089573
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Time series pre-training has recently garnered wide attention for its potential to reduce labeling expenses and benefit various downstream tasks. Prior methods are mainly based on pre-training techniques well-acknowledged in vision or language, such as masked modeling and contrastive learning. However, randomly masking time series or calculating series-wise similarity will distort or neglect inherent temporal correlations crucial in time series data. To emphasize temporal correlation modeling, this paper proposes TimeSiam as a simple but effective self-supervised pre-training framework for Time series based on Siamese networks. Concretely, TimeSiam pre-trains Siamese encoders to capture intrinsic temporal correlations between randomly sampled past and current subseries. With a simple data augmentation method (e.g.~masking), TimeSiam can benefit from diverse augmented subseries and learn internal time-dependent representations through a past-to-current reconstruction. Moreover, learnable lineage embeddings are also introduced to distinguish temporal distance between sampled series and further foster the learning of diverse temporal correlations. TimeSiam consistently outperforms extensive advanced pre-training baselines, demonstrating superior forecasting and classification capabilities across 13 standard benchmarks in both intra- and cross-domain scenarios.

Related papers

• Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning [22.28251586213348]
  aLLM4TS is an innovative framework that adapts Large Language Models (LLMs) for time-series representation learning. A distinctive element of our framework is the patch-wise decoding layer, which departs from previous methods reliant on sequence-level decoding.
  arXiv  Detail & Related papers  (2024-02-07T13:51:26Z)
• Unified Training of Universal Time Series Forecasting Transformers [104.56318980466742]
  We present a Masked-based Universal Time Series Forecasting Transformer (Moirai) Moirai is trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains. Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models.
  arXiv  Detail & Related papers  (2024-02-04T20:00:45Z)
• SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling [82.69579113377192]
  SimMTM is a simple pre-training framework for Masked Time-series Modeling. SimMTM recovers masked time points by the weighted aggregation of multiple neighbors outside the manifold. SimMTM achieves state-of-the-art fine-tuning performance compared to the most advanced time series pre-training methods.
  arXiv  Detail & Related papers  (2023-02-02T04:12:29Z)
• Ti-MAE: Self-Supervised Masked Time Series Autoencoders [16.98069693152999]
  We propose a novel framework named Ti-MAE, in which the input time series are assumed to follow an integrate distribution. Ti-MAE randomly masks out embedded time series data and learns an autoencoder to reconstruct them at the point-level. Experiments on several public real-world datasets demonstrate that our framework of masked autoencoding could learn strong representations directly from the raw data.
  arXiv  Detail & Related papers  (2023-01-21T03:20:23Z)
• Multi-scale Attention Flow for Probabilistic Time Series Forecasting [68.20798558048678]
  We propose a novel non-autoregressive deep learning model, called Multi-scale Attention Normalizing Flow(MANF) Our model avoids the influence of cumulative error and does not increase the time complexity. Our model achieves state-of-the-art performance on many popular multivariate datasets.
  arXiv  Detail & Related papers  (2022-05-16T07:53:42Z)
• Cluster-and-Conquer: A Framework For Time-Series Forecasting [94.63501563413725]
  We propose a three-stage framework for forecasting high-dimensional time-series data. Our framework is highly general, allowing for any time-series forecasting and clustering method to be used in each step. When instantiated with simple linear autoregressive models, we are able to achieve state-of-the-art results on several benchmark datasets.
  arXiv  Detail & Related papers  (2021-10-26T20:41:19Z)
• Synergetic Learning of Heterogeneous Temporal Sequences for Multi-Horizon Probabilistic Forecasting [48.8617204809538]
  We propose Variational Synergetic Multi-Horizon Network (VSMHN), a novel deep conditional generative model. To learn complex correlations across heterogeneous sequences, a tailored encoder is devised to combine the advances in deep point processes models and variational recurrent neural networks. Our model can be trained effectively using variational inference and generates predictions with Monte-Carlo simulation.
  arXiv  Detail & Related papers  (2021-01-31T11:00:55Z)
• Series Saliency: Temporal Interpretation for Multivariate Time Series Forecasting [30.054015098590874]
  We present the series saliency framework for temporal interpretation for time series forecasting. By extracting the "series images" from the sliding windows of the time series, we apply the saliency map segmentation. Our framework generates temporal interpretations for the time series forecasting task while produces accurate time series forecast.
  arXiv  Detail & Related papers  (2020-12-16T23:48:00Z)
• Self-Supervised Time Series Representation Learning by Inter-Intra Relational Reasoning [18.72937677485634]
  We present SelfTime: a general self-supervised time series representation learning framework. We explore the inter-sample relation and intra-temporal relation of time series to learn the underlying structure feature on the unlabeled time series. The useful representations of time series are extracted from the backbone under the supervision of relation reasoning heads.
  arXiv  Detail & Related papers  (2020-11-27T04:04:17Z)
• A Deep Structural Model for Analyzing Correlated Multivariate Time Series [11.009809732645888]
  We present a deep learning structural time series model which can handle correlated multivariate time series input. The model explicitly learns/extracts the trend, seasonality, and event components. We compare our model with several state-of-the-art methods through a comprehensive set of experiments on a variety of time series data sets.
  arXiv  Detail & Related papers  (2020-01-02T18:48:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.