Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization

• URL: http://arxiv.org/abs/2412.05244v1
• Date: Fri, 06 Dec 2024 18:22:59 GMT
• Title: Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
• Authors: Luca Masserano, Abdul Fatir Ansari, Boran Han, Xiyuan Zhang, Christos Faloutsos, Michael W. Mahoney, Andrew Gordon Wilson, Youngsuk Park, Syama Rangapuram, Danielle C. Maddix, Yuyang Wang,
• Abstract summary: We develop a wavelet-based tokenizer that allows models to learn complex representations directly in the space of time-localized frequencies. Our method first scales and decomposes the input time series, then thresholds and quantizes the wavelet coefficients, and finally pre-trains an autoregressive model to forecast coefficients for the forecast horizon.
• Score: 74.3339999119713
• License:
• Abstract: How to best develop foundational models for time series forecasting remains an important open question. Tokenization is a crucial consideration in this effort: what is an effective discrete vocabulary for a real-valued sequential input? To address this question, we develop WaveToken, a wavelet-based tokenizer that allows models to learn complex representations directly in the space of time-localized frequencies. Our method first scales and decomposes the input time series, then thresholds and quantizes the wavelet coefficients, and finally pre-trains an autoregressive model to forecast coefficients for the forecast horizon. By decomposing coarse and fine structures in the inputs, wavelets provide an eloquent and compact language for time series forecasting that simplifies learning. Empirical results on a comprehensive benchmark, including 42 datasets for both in-domain and zero-shot settings, show that WaveToken: i) provides better accuracy than recently proposed foundation models for forecasting while using a much smaller vocabulary (1024 tokens), and performs on par or better than modern deep learning models trained specifically on each dataset; and ii) exhibits superior generalization capabilities, achieving the best average rank across all datasets for three complementary metrics. In addition, we show that our method can easily capture complex temporal patterns of practical relevance that are challenging for other recent pre-trained models, including trends, sparse spikes, and non-stationary time series with varying frequencies evolving over time.

Related papers

• Sundial: A Family of Highly Capable Time Series Foundation Models [64.6322079384575]
  We introduce Sundial, a family of native, flexible, and scalable time series foundation models. Our model is pre-trained without specifying any prior distribution and can generate multiple probable predictions. By mitigating mode collapse through TimeFlow Loss, we pre-train a family of Sundial models on TimeBench, which exhibit unprecedented model capacity and generalization performance.
  arXiv  Detail & Related papers  (2025-02-02T14:52:50Z)
• Beyond Data Scarcity: A Frequency-Driven Framework for Zero-Shot Forecasting [15.431513584239047]
  Time series forecasting is critical in numerous real-world applications. Traditional forecasting techniques struggle when data is scarce or not available at all. Recent advancements often leverage large-scale foundation models for such tasks.
  arXiv  Detail & Related papers  (2024-11-24T07:44:39Z)
• Chronos: Learning the Language of Time Series [79.38691251254173]
  Chronos is a framework for pretrained probabilistic time series models. We show that Chronos models can leverage time series data from diverse domains to improve zero-shot accuracy on unseen forecasting tasks.
  arXiv  Detail & Related papers  (2024-03-12T16:53:54Z)
• Unified Training of Universal Time Series Forecasting Transformers [104.56318980466742]
  We present a Masked-based Universal Time Series Forecasting Transformer (Moirai) Moirai is trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains. Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models.
  arXiv  Detail & Related papers  (2024-02-04T20:00:45Z)
• Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain [54.67888148566323]
  We introduce three large-scale time series forecasting datasets from the cloud operations domain. We show it is a strong zero-shot baseline and benefits from further scaling, both in model and dataset size. Accompanying these datasets and results is a suite of comprehensive benchmark results comparing classical and deep learning baselines to our pre-trained method.
  arXiv  Detail & Related papers  (2023-10-08T08:09:51Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)
• Split Time Series into Patches: Rethinking Long-term Series Forecasting with Dateformer [17.454822366228335]
  Time is one of the most significant characteristics of time-series, yet has received insufficient attention. We propose Dateformer who turns attention to modeling time instead of following the above practice. Dateformer yields state-of-the-art accuracy with a 40% remarkable relative improvement, and broadens the maximum credible forecasting range to a half-yearly level.
  arXiv  Detail & Related papers  (2022-07-12T08:58:44Z)
• PRNet: A Periodic Residual Learning Network for Crowd Flow Forecasting [8.50942649992681]
  We devise a novel periodic residual learning network (PRNet) for better modeling the periodicity in crowd flow data. PRNet frames the crowd flow forecasting as a periodic residual learning problem by modeling the deviation between the input (the previous time period) and the output (the future time period) Experimental results on two real-world datasets demonstrate that PRNet outperforms the state-of-the-art methods in terms of both accuracy and robustness.
  arXiv  Detail & Related papers  (2021-12-08T12:04:27Z)
• Improved Predictive Deep Temporal Neural Networks with Trend Filtering [22.352437268596674]
  We propose a new prediction framework based on deep neural networks and a trend filtering. We reveal that the predictive performance of deep temporal neural networks improves when the training data is temporally processed by a trend filtering.
  arXiv  Detail & Related papers  (2020-10-16T08:29:36Z)
• Conditional Mutual information-based Contrastive Loss for Financial Time Series Forecasting [12.0855096102517]
  We present a representation learning framework for financial time series forecasting. In this paper, we propose to first learn compact representations from time series data, then use the learned representations to train a simpler model for predicting time series movements.
  arXiv  Detail & Related papers  (2020-02-18T15:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.