Exploring Progress in Multivariate Time Series Forecasting:
Comprehensive Benchmarking and Heterogeneity Analysis
- URL: http://arxiv.org/abs/2310.06119v1
- Date: Mon, 9 Oct 2023 19:52:22 GMT
- Title: Exploring Progress in Multivariate Time Series Forecasting:
Comprehensive Benchmarking and Heterogeneity Analysis
- Authors: Zezhi Shao, Fei Wang, Yongjun Xu, Wei Wei, Chengqing Yu, Zhao Zhang,
Di Yao, Guangyin Jin, Xin Cao, Gao Cong, Christian S. Jensen, Xueqi Cheng
- Abstract summary: We introduce BasicTS, a benchmark designed for fair comparisons in MTS forecasting.
We highlight the heterogeneity among MTS datasets and classify them based on temporal and spatial characteristics.
- Score: 72.18987459587682
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Multivariate Time Series (MTS) widely exists in real-word complex systems,
such as traffic and energy systems, making their forecasting crucial for
understanding and influencing these systems. Recently, deep learning-based
approaches have gained much popularity for effectively modeling temporal and
spatial dependencies in MTS, specifically in Long-term Time Series Forecasting
(LTSF) and Spatial-Temporal Forecasting (STF). However, the fair benchmarking
issue and the choice of technical approaches have been hotly debated in related
work. Such controversies significantly hinder our understanding of progress in
this field. Thus, this paper aims to address these controversies to present
insights into advancements achieved. To resolve benchmarking issues, we
introduce BasicTS, a benchmark designed for fair comparisons in MTS
forecasting. BasicTS establishes a unified training pipeline and reasonable
evaluation settings, enabling an unbiased evaluation of over 30 popular MTS
forecasting models on more than 18 datasets. Furthermore, we highlight the
heterogeneity among MTS datasets and classify them based on temporal and
spatial characteristics. We further prove that neglecting heterogeneity is the
primary reason for generating controversies in technical approaches. Moreover,
based on the proposed BasicTS and rich heterogeneous MTS datasets, we conduct
an exhaustive and reproducible performance and efficiency comparison of popular
models, providing insights for researchers in selecting and designing MTS
forecasting models.
Related papers
- Understanding Different Design Choices in Training Large Time Series Models [71.20102277299445]
Training Large Time Series Models (LTSMs) on heterogeneous time series data poses unique challenges.
We propose emphtime series prompt, a novel statistical prompting strategy tailored to time series data.
We introduce textttLTSM-bundle, which bundles the best design choices we have identified.
arXiv Detail & Related papers (2024-06-20T07:09:19Z) - GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models [56.63218531256961]
We introduce GenBench, a benchmarking suite specifically tailored for evaluating the efficacy of Genomic Foundation Models.
GenBench offers a modular and expandable framework that encapsulates a variety of state-of-the-art methodologies.
We provide a nuanced analysis of the interplay between model architecture and dataset characteristics on task-specific performance.
arXiv Detail & Related papers (2024-06-01T08:01:05Z) - Multi-Modality Spatio-Temporal Forecasting via Self-Supervised Learning [11.19088022423885]
We propose a novel MoST learning framework via Self-Supervised Learning, namely MoSSL.
Results on two real-world MoST datasets verify the superiority of our approach compared with the state-of-the-art baselines.
arXiv Detail & Related papers (2024-05-06T08:24:06Z) - Embedded feature selection in LSTM networks with multi-objective
evolutionary ensemble learning for time series forecasting [49.1574468325115]
We present a novel feature selection method embedded in Long Short-Term Memory networks.
Our approach optimize the weights and biases of the LSTM in a partitioned manner.
Experimental evaluations on air quality time series data from Italy and southeast Spain demonstrate that our method substantially improves the ability generalization of conventional LSTMs.
arXiv Detail & Related papers (2023-12-29T08:42:10Z) - TRAM: Benchmarking Temporal Reasoning for Large Language Models [12.112914393948415]
We introduce TRAM, a temporal reasoning benchmark composed of ten datasets.
We evaluate popular language models like GPT-4 and Llama2 in zero-shot and few-shot scenarios.
Our findings indicate that the best-performing model lags significantly behind human performance.
arXiv Detail & Related papers (2023-10-02T00:59:07Z) - OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive
Learning [67.07363529640784]
We propose OpenSTL to categorize prevalent approaches into recurrent-based and recurrent-free models.
We conduct standard evaluations on datasets across various domains, including synthetic moving object trajectory, human motion, driving scenes, traffic flow and forecasting weather.
We find that recurrent-free models achieve a good balance between efficiency and performance than recurrent models.
arXiv Detail & Related papers (2023-06-20T03:02:14Z) - Disentangling Structured Components: Towards Adaptive, Interpretable and
Scalable Time Series Forecasting [52.47493322446537]
We develop a adaptive, interpretable and scalable forecasting framework, which seeks to individually model each component of the spatial-temporal patterns.
SCNN works with a pre-defined generative process of MTS, which arithmetically characterizes the latent structure of the spatial-temporal patterns.
Extensive experiments are conducted to demonstrate that SCNN can achieve superior performance over state-of-the-art models on three real-world datasets.
arXiv Detail & Related papers (2023-05-22T13:39:44Z) - MTS-Mixers: Multivariate Time Series Forecasting via Factorized Temporal
and Channel Mixing [18.058617044421293]
This paper investigates the contributions and deficiencies of attention mechanisms on the performance of time series forecasting.
We propose MTS-Mixers, which use two factorized modules to capture temporal and channel dependencies.
Experimental results on several real-world datasets show that MTS-Mixers outperform existing Transformer-based models with higher efficiency.
arXiv Detail & Related papers (2023-02-09T08:52:49Z) - Spatial-Temporal Identity: A Simple yet Effective Baseline for
Multivariate Time Series Forecasting [17.84296081495185]
We explore the critical factors of MTS forecasting and design a model that is as powerful as STGNNs, but more concise and efficient.
We identify the indistinguishability of samples in both spatial and temporal dimensions as a key bottleneck, and propose a simple yet effective baseline for MTS forecasting.
These results suggest that we can design efficient and effective models as long as they solve the indistinguishability of samples, without being limited to STGNNs.
arXiv Detail & Related papers (2022-08-10T09:25:43Z) - Analysis of Empirical Mode Decomposition-based Load and Renewable Time
Series Forecasting [1.1602089225841632]
Time series (TS) related to historical load and renewable generation are decomposed into intrinsic mode functions (IMFs)
The method is prone to several issues, including modal aliasing and boundary effect problems.
Underestimating these issues can lead to poor performance of the forecast model in real-time applications.
arXiv Detail & Related papers (2020-11-23T14:08:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.