AutoTimes: Autoregressive Time Series Forecasters via Large Language Models

• URL: http://arxiv.org/abs/2402.02370v4
• Date: Thu, 31 Oct 2024 11:37:41 GMT
• Title: AutoTimes: Autoregressive Time Series Forecasters via Large Language Models
• Authors: Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, Mingsheng Long,
• Abstract summary: AutoTimes projects time series into the embedding space of language tokens and autoregressively generates future predictions with arbitrary lengths. We formulate time series as prompts, extending the context for prediction beyond the lookback window. AutoTimes achieves state-of-the-art with 0.1% trainable parameters and over $5times$ training/inference speedup.
• Score: 67.83502953961505
• License:
• Abstract: Foundation models of time series have not been fully developed due to the limited availability of time series corpora and the underexploration of scalable pre-training. Based on the similar sequential formulation of time series and natural language, increasing research demonstrates the feasibility of leveraging large language models (LLM) for time series. Nevertheless, the inherent autoregressive property and decoder-only architecture of LLMs have not been fully considered, resulting in insufficient utilization of LLM abilities. To fully revitalize the general-purpose token transition and multi-step generation capability of large language models, we propose AutoTimes to repurpose LLMs as autoregressive time series forecasters, which projects time series into the embedding space of language tokens and autoregressively generates future predictions with arbitrary lengths. Compatible with any decoder-only LLMs, the consequent forecaster exhibits the flexibility of the lookback length and scalability with larger LLMs. Further, we formulate time series as prompts, extending the context for prediction beyond the lookback window, termed in-context forecasting. By introducing LLM-embedded textual timestamps, AutoTimes can utilize chronological information to align multivariate time series. Empirically, AutoTimes achieves state-of-the-art with 0.1% trainable parameters and over $5\times$ training/inference speedup compared to advanced LLM-based forecasters. Code is available at this repository: https://github.com/thuml/AutoTimes.

Related papers

• Towards Time Series Reasoning with LLMs [0.4369058206183195]
  We propose a novel multi-modal time-series LLM approach that learns generalizable information across various domains with powerful zero-shot performance. We show that our model learns a latent representation that reflects specific time-series features, as well as outperforming GPT-4o on a set of zero-shot reasoning tasks.
  arXiv  Detail & Related papers  (2024-09-17T17:23:44Z)
• Are Language Models Actually Useful for Time Series Forecasting? [21.378728572776897]
  We find that removing the LLM component or replacing it with a basic attention layer does not degrade forecasting performance. We also find that despite their significant computational cost, pretrained LLMs do no better than models trained from scratch. We explore time series encoders and find that patching and attention structures perform similarly to LLM-based forecasters.
  arXiv  Detail & Related papers  (2024-06-22T03:33:38Z)
• TimeCMA: Towards LLM-Empowered Time Series Forecasting via Cross-Modality Alignment [21.690191536424567]
  TimeCMA is a framework for time series forecasting with cross-modality alignment. Extensive experiments on real data offer insight into the accuracy and efficiency of the proposed framework.
  arXiv  Detail & Related papers  (2024-06-03T00:27:29Z)
• Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilities [46.02234423159257]
  Large language models (LLMs) have been applied in many fields and have developed rapidly in recent years. Recent works treat large language models as emphzero-shot time series reasoners without further fine-tuning. Our study shows that LLMs perform well in predicting time series with clear patterns and trends, but face challenges with datasets lacking periodicity.
  arXiv  Detail & Related papers  (2024-02-16T17:15:28Z)
• Unified Training of Universal Time Series Forecasting Transformers [104.56318980466742]
  We present a Masked-based Universal Time Series Forecasting Transformer (Moirai) Moirai is trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains. Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models.
  arXiv  Detail & Related papers  (2024-02-04T20:00:45Z)
• Timer: Generative Pre-trained Transformers Are Large Time Series Models [83.03091523806668]
  This paper aims at the early development of large time series models (LTSM) During pre-training, we curate large-scale datasets with up to 1 billion time points. To meet diverse application needs, we convert forecasting, imputation, and anomaly detection of time series into a unified generative task.
  arXiv  Detail & Related papers  (2024-02-04T06:55:55Z)
• Large Language Models Are Zero-Shot Time Series Forecasters [48.73953666153385]
  By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. We find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero-shot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks.
  arXiv  Detail & Related papers  (2023-10-11T19:01:28Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.