HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling for
Long-Term Forecasting
- URL: http://arxiv.org/abs/2401.05012v1
- Date: Wed, 10 Jan 2024 09:00:03 GMT
- Title: HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling for
Long-Term Forecasting
- Authors: Shubao Zhao, Ming Jin, Zhaoxiang Hou, Chengyi Yang, Zengxiang Li,
Qingsong Wen, Yi Wang
- Abstract summary: HiMTM is a hierarchical multi-scale masked time series modeling method designed for long-term forecasting.
It comprises four integral components: (1) hierarchical multi-scale transformer (HMT) to capture temporal information at different scales; (2) decoupled encoder-decoder (DED) forces the encoder to focus on feature extraction; while the decoder to focus on pretext tasks.
We conduct extensive experiments on 7 mainstream datasets to prove that HiMTM has obvious advantages over contemporary self-supervised and end-to-end learning methods.
- Score: 18.59792043113792
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Time series forecasting is crucial and challenging in the real world. The
recent surge in interest regarding time series foundation models, which cater
to a diverse array of downstream tasks, is noteworthy. However, existing
methods often overlook the multi-scale nature of time series, an aspect crucial
for precise forecasting. To bridge this gap, we propose HiMTM, a hierarchical
multi-scale masked time series modeling method designed for long-term
forecasting. Specifically, it comprises four integral components: (1)
hierarchical multi-scale transformer (HMT) to capture temporal information at
different scales; (2) decoupled encoder-decoder (DED) forces the encoder to
focus on feature extraction, while the decoder to focus on pretext tasks; (3)
multi-scale masked reconstruction (MMR) provides multi-stage supervision
signals for pre-training; (4) cross-scale attention fine-tuning (CSA-FT) to
capture dependencies between different scales for forecasting. Collectively,
these components enhance multi-scale feature extraction capabilities in masked
time series modeling and contribute to improved prediction accuracy. We conduct
extensive experiments on 7 mainstream datasets to prove that HiMTM has obvious
advantages over contemporary self-supervised and end-to-end learning methods.
The effectiveness of HiMTM is further showcased by its application in the
industry of natural gas demand forecasting.
Related papers
- MGCP: A Multi-Grained Correlation based Prediction Network for Multivariate Time Series [54.91026286579748]
We propose a Multi-Grained Correlations-based Prediction Network.
It simultaneously considers correlations at three levels to enhance prediction performance.
It employs adversarial training with an attention mechanism-based predictor and conditional discriminator to optimize prediction results at coarse-grained level.
arXiv Detail & Related papers (2024-05-30T03:32:44Z) - Multi-Patch Prediction: Adapting LLMs for Time Series Representation
Learning [22.28251586213348]
aLLM4TS is an innovative framework that adapts Large Language Models (LLMs) for time-series representation learning.
A distinctive element of our framework is the patch-wise decoding layer, which departs from previous methods reliant on sequence-level decoding.
arXiv Detail & Related papers (2024-02-07T13:51:26Z) - Unified Training of Universal Time Series Forecasting Transformers [104.56318980466742]
We present a Masked-based Universal Time Series Forecasting Transformer (Moirai)
Moirai is trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains.
Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models.
arXiv Detail & Related papers (2024-02-04T20:00:45Z) - Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
Time series forecasting holds significant importance in many real-world dynamic systems.
We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting.
Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
arXiv Detail & Related papers (2023-10-03T01:31:25Z) - Revealing the Power of Spatial-Temporal Masked Autoencoders in
Multivariate Time Series Forecasting [17.911251232225094]
We propose an MTS forecasting framework that leverages masked autoencoders to enhance the performance of spatial-temporal baseline models.
In the pretraining stage, an encoder-decoder architecture is employed to process partially visible MTS data.
In the fine-tuning stage, the encoder is retained, and the original decoder from existing spatial-temporal models is appended for forecasting.
arXiv Detail & Related papers (2023-09-26T18:05:19Z) - FormerTime: Hierarchical Multi-Scale Representations for Multivariate
Time Series Classification [53.55504611255664]
FormerTime is a hierarchical representation model for improving the classification capacity for the multivariate time series classification task.
It exhibits three aspects of merits: (1) learning hierarchical multi-scale representations from time series data, (2) inheriting the strength of both transformers and convolutional networks, and (3) tacking the efficiency challenges incurred by the self-attention mechanism.
arXiv Detail & Related papers (2023-02-20T07:46:14Z) - MTS-Mixers: Multivariate Time Series Forecasting via Factorized Temporal
and Channel Mixing [18.058617044421293]
This paper investigates the contributions and deficiencies of attention mechanisms on the performance of time series forecasting.
We propose MTS-Mixers, which use two factorized modules to capture temporal and channel dependencies.
Experimental results on several real-world datasets show that MTS-Mixers outperform existing Transformer-based models with higher efficiency.
arXiv Detail & Related papers (2023-02-09T08:52:49Z) - SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling [82.69579113377192]
SimMTM is a simple pre-training framework for Masked Time-series Modeling.
SimMTM recovers masked time points by the weighted aggregation of multiple neighbors outside the manifold.
SimMTM achieves state-of-the-art fine-tuning performance compared to the most advanced time series pre-training methods.
arXiv Detail & Related papers (2023-02-02T04:12:29Z) - Ti-MAE: Self-Supervised Masked Time Series Autoencoders [16.98069693152999]
We propose a novel framework named Ti-MAE, in which the input time series are assumed to follow an integrate distribution.
Ti-MAE randomly masks out embedded time series data and learns an autoencoder to reconstruct them at the point-level.
Experiments on several public real-world datasets demonstrate that our framework of masked autoencoding could learn strong representations directly from the raw data.
arXiv Detail & Related papers (2023-01-21T03:20:23Z) - An Unsupervised Short- and Long-Term Mask Representation for
Multivariate Time Series Anomaly Detection [2.387411589813086]
This paper proposes an anomaly detection method based on unsupervised Short- and Long-term Mask Representation learning (SLMR)
Experiments show that the performance of our method outperforms other state-of-the-art models on three real-world datasets.
arXiv Detail & Related papers (2022-08-19T09:34:11Z) - Multi-scale Attention Flow for Probabilistic Time Series Forecasting [68.20798558048678]
We propose a novel non-autoregressive deep learning model, called Multi-scale Attention Normalizing Flow(MANF)
Our model avoids the influence of cumulative error and does not increase the time complexity.
Our model achieves state-of-the-art performance on many popular multivariate datasets.
arXiv Detail & Related papers (2022-05-16T07:53:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.