TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation

• URL: http://arxiv.org/abs/2410.05711v4
• Date: Fri, 21 Feb 2025 03:00:55 GMT
• Title: TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation
• Authors: Daoyu Wang, Mingyue Cheng, Zhiding Liu, Qi Liu, Enhong Chen,
• Abstract summary: We propose TimeDART, a novel self-supervised time series pre-training framework.<n>TimeDART unifies two powerful generative paradigms to learn more transferable representations.<n>We conduct extensive experiments on public datasets for time series forecasting and classification.
• Score: 47.58016750718323
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised learning has garnered increasing attention in time series analysis for benefiting various downstream tasks and reducing reliance on labeled data. Despite its effectiveness, existing methods often struggle to comprehensively capture both long-term dynamic evolution and subtle local patterns in a unified manner. In this work, we propose TimeDART, a novel self-supervised time series pre-training framework that unifies two powerful generative paradigms to learn more transferable representations. Specifically, we first employ a causal Transformer encoder, accompanied by a patch-based embedding strategy, to model the evolving trends from left to right. Building on this global modeling, we further introduce a denoising diffusion process to capture fine-grained local patterns through forward diffusion and reverse denoising. Finally, we optimize the model in an autoregressive manner. As a result, TimeDART effectively accounts for both global and local sequence features in a coherent way. We conduct extensive experiments on public datasets for time series forecasting and classification. The experimental results demonstrate that TimeDART consistently outperforms previous compared methods, validating the effectiveness of our approach. Our code is available at https://github.com/Melmaphother/TimeDART.

Related papers

• Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Models [71.63194926457119]
  We introduce Dynamical Diffusion (DyDiff), a theoretically sound framework that incorporates temporally aware forward and reverse processes. Experiments across scientifictemporal forecasting, video prediction, and time series forecasting demonstrate that Dynamical Diffusion consistently improves performance in temporal predictive tasks.
  arXiv  Detail & Related papers  (2025-03-02T16:10:32Z)
• Understanding the Limits of Deep Tabular Methods with Temporal Shift [28.738848567072004]
  We introduce a plug-and-play temporal embedding method based on Fourier series expansion to learn and incorporate temporal patterns. Our experiments demonstrate that this temporal embedding, combined with the improved training protocol, provides a more effective and robust framework for learning from temporal data.
  arXiv  Detail & Related papers  (2025-02-27T16:48:53Z)
• Generative Regression Based Watch Time Prediction for Short-Video Recommendation [36.95095097454143]
  Watch time prediction has emerged as a pivotal task in short video recommendation systems. Recent studies have attempted to address these issues by converting the continuous watch time estimation into an ordinal regression task. We propose a novel Generative Regression (GR) framework that reformulates WTP as a sequence generation task.
  arXiv  Detail & Related papers  (2024-12-28T16:48:55Z)
• WaveGNN: Modeling Irregular Multivariate Time Series for Accurate Predictions [3.489870763747715]
  Real-world time series often exhibit irregularities such as misaligned timestamps, missing entries, and variable sampling rates. Existing approaches often rely on imputation, which can introduce biases. We present WaveGNN, a novel framework designed to embed irregularly sampled time series data for accurate predictions.
  arXiv  Detail & Related papers  (2024-12-14T00:03:44Z)
• TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model [11.281386703572842]
  A family of models have been developed, utilizing a temporal auto-regressive generative Transformer architecture. TimeDiT is a general foundation model for time series that employs a denoising diffusion paradigm instead of temporal auto-regressive generation. Extensive experiments conducted on a varity of tasks such as forecasting, imputation, and anomaly detection, demonstrate the effectiveness of TimeDiT.
  arXiv  Detail & Related papers  (2024-09-03T22:31:57Z)
• sTransformer: A Modular Approach for Extracting Inter-Sequential and Temporal Information for Time-Series Forecasting [6.434378359932152]
  We review and categorize existing Transformer-based models into two main types: (1) modifications to the model structure and (2) modifications to the input data. We propose $textbfsTransformer$, which introduces the Sequence and Temporal Convolutional Network (STCN) to fully capture both sequential and temporal information. We compare our model with linear models and existing forecasting models on long-term time-series forecasting, achieving new state-of-the-art results.
  arXiv  Detail & Related papers  (2024-08-19T06:23:41Z)
• Attention as Robust Representation for Time Series Forecasting [23.292260325891032]
  Time series forecasting is essential for many practical applications. Transformers' key feature, the attention mechanism, dynamically fusing embeddings to enhance data representation, often relegating attention weights to a byproduct role. Our approach elevates attention weights as the primary representation for time series, capitalizing on the temporal relationships among data points to improve forecasting accuracy.
  arXiv  Detail & Related papers  (2024-02-08T03:00:50Z)
• Unified Training of Universal Time Series Forecasting Transformers [104.56318980466742]
  We present a Masked-based Universal Time Series Forecasting Transformer (Moirai) Moirai is trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains. Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models.
  arXiv  Detail & Related papers  (2024-02-04T20:00:45Z)
• TimeGraphs: Graph-based Temporal Reasoning [64.18083371645956]
  TimeGraphs is a novel approach that characterizes dynamic interactions as a hierarchical temporal graph. Our approach models the interactions using a compact graph-based representation, enabling adaptive reasoning across diverse time scales. We evaluate TimeGraphs on multiple datasets with complex, dynamic agent interactions, including a football simulator, the Resistance game, and the MOMA human activity dataset.
  arXiv  Detail & Related papers  (2024-01-06T06:26:49Z)
• Generative Modeling of Regular and Irregular Time Series Data via Koopman VAEs [50.25683648762602]
  We introduce Koopman VAE, a new generative framework that is based on a novel design for the model prior. Inspired by Koopman theory, we represent the latent conditional prior dynamics using a linear map. KoVAE outperforms state-of-the-art GAN and VAE methods across several challenging synthetic and real-world time series generation benchmarks.
  arXiv  Detail & Related papers  (2023-10-04T07:14:43Z)
• TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders [55.00904795497786]
  We propose TimeMAE, a novel self-supervised paradigm for learning transferrable time series representations based on transformer networks. The TimeMAE learns enriched contextual representations of time series with a bidirectional encoding scheme. To solve the discrepancy issue incurred by newly injected masked embeddings, we design a decoupled autoencoder architecture.
  arXiv  Detail & Related papers  (2023-03-01T08:33:16Z)
• Ti-MAE: Self-Supervised Masked Time Series Autoencoders [16.98069693152999]
  We propose a novel framework named Ti-MAE, in which the input time series are assumed to follow an integrate distribution. Ti-MAE randomly masks out embedded time series data and learns an autoencoder to reconstruct them at the point-level. Experiments on several public real-world datasets demonstrate that our framework of masked autoencoding could learn strong representations directly from the raw data.
  arXiv  Detail & Related papers  (2023-01-21T03:20:23Z)
• Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement [51.55157852647306]
  Time series forecasting has been a widely explored task of great importance in many applications. It is common that real-world time series data are recorded in a short time period, which results in a big gap between the deep model and the limited and noisy time series. We propose to address the time series forecasting problem with generative modeling and propose a bidirectional variational auto-encoder equipped with diffusion, denoise, and disentanglement.
  arXiv  Detail & Related papers  (2023-01-08T12:20:46Z)
• Towards Long-Term Time-Series Forecasting: Feature, Pattern, and Distribution [57.71199089609161]
  Long-term time-series forecasting (LTTF) has become a pressing demand in many applications, such as wind power supply planning. Transformer models have been adopted to deliver high prediction capacity because of the high computational self-attention mechanism. We propose an efficient Transformerbased model, named Conformer, which differentiates itself from existing methods for LTTF in three aspects.
  arXiv  Detail & Related papers  (2023-01-05T13:59:29Z)
• Gait Recognition in the Wild with Multi-hop Temporal Switch [81.35245014397759]
  gait recognition in the wild is a more practical problem that has attracted the attention of the community of multimedia and computer vision. This paper presents a novel multi-hop temporal switch method to achieve effective temporal modeling of gait patterns in real-world scenes.
  arXiv  Detail & Related papers  (2022-09-01T10:46:09Z)
• Large Scale Time-Series Representation Learning via Simultaneous Low and High Frequency Feature Bootstrapping [7.0064929761691745]
  We propose a non-contrastive self-supervised learning approach efficiently captures low and high-frequency time-varying features. Our method takes raw time series data as input and creates two different augmented views for two branches of the model. To demonstrate the robustness of our model we performed extensive experiments and ablation studies on five real-world time-series datasets.
  arXiv  Detail & Related papers  (2022-04-24T14:39:47Z)
• Stacking VAE with Graph Neural Networks for Effective and Interpretable Time Series Anomaly Detection [5.935707085640394]
  We propose a stacking variational auto-encoder (VAE) model with graph neural networks for the effective and interpretable time-series anomaly detection. We show that our proposed model outperforms the strong baselines on three public datasets with considerable improvements.
  arXiv  Detail & Related papers  (2021-05-18T09:50:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.