TimeCMA: Towards LLM-Empowered Time Series Forecasting via Cross-Modality Alignment

• URL: http://arxiv.org/abs/2406.01638v3
• Date: Fri, 14 Jun 2024 01:39:29 GMT
• Title: TimeCMA: Towards LLM-Empowered Time Series Forecasting via Cross-Modality Alignment
• Authors: Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang, Lingzheng Zhang, Cheng Long, Ziyue Li, Rui Zhao,
• Abstract summary: TimeCMA is a framework for time series forecasting with cross-modality alignment. Extensive experiments on real data offer insight into the accuracy and efficiency of the proposed framework.
• Score: 21.690191536424567
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The widespread adoption of scalable mobile sensing has led to large amounts of time series data for real-world applications. A fundamental application is multivariate time series forecasting (MTSF), which aims to predict future time series values based on historical observations. Existing MTSF methods suffer from limited parameterization and small-scale training data. Recently, Large language models (LLMs) have been introduced in time series, which achieve promising forecasting performance but incur heavy computational costs. To solve these challenges, we propose TimeCMA, an LLM-empowered framework for time series forecasting with cross-modality alignment. We design a dual-modality encoding module with two branches, where the time series encoding branch extracts relatively low-quality yet pure embeddings of time series through an inverted Transformer. In addition, the LLM-empowered encoding branch wraps the same time series as prompts to obtain high-quality yet entangled prompt embeddings via a Pre-trained LLM. Then, we design a cross-modality alignment module to retrieve high-quality and pure time series embeddings from the prompt embeddings. Moreover, we develop a time series forecasting module to decode the aligned embeddings while capturing dependencies among multiple variables for forecasting. Notably, we tailor the prompt to encode sufficient temporal information into a last token and design the last token embedding storage to reduce computational costs. Extensive experiments on real data offer insight into the accuracy and efficiency of the proposed framework.

Related papers

• Adapting Large Language Models for Time Series Modeling via a Novel Parameter-efficient Adaptation Method [9.412920379798928]
  Time series modeling holds significant importance in many real-world applications. We propose the Time-LlaMA framework to align the time series and natural language modalities. We show that our proposed method achieves the state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2025-02-19T13:52:26Z)
• TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agents [52.13094810313054]
  TimeCAP is a time-series processing framework that creatively employs Large Language Models (LLMs) as contextualizers of time series data. TimeCAP incorporates two independent LLM agents: one generates a textual summary capturing the context of the time series, while the other uses this enriched summary to make more informed predictions. Experimental results on real-world datasets demonstrate that TimeCAP outperforms state-of-the-art methods for time series event prediction.
  arXiv  Detail & Related papers  (2025-02-17T04:17:27Z)
• Rethinking Time Series Forecasting with LLMs via Nearest Neighbor Contrastive Learning [1.7892194562398749]
  We propose NNCL-TLLM: Nearest Neighbor Contrastive Learning for Time series forecasting via Large Language Models. First, we generate time series compatible text prototypes such that each text prototype represents both word token embeddings in its neighborhood and time series characteristics. We then fine-tune the layer normalization and positional embeddings of the LLM, keeping the other layers intact, reducing the trainable parameters and decreasing the computational cost.
  arXiv  Detail & Related papers  (2024-12-06T06:32:47Z)
• Hierarchical Multimodal LLMs with Semantic Space Alignment for Enhanced Time Series Classification [4.5939667818289385]
  HiTime is a hierarchical multi-modal model that seamlessly integrates temporal information into large language models. Our findings highlight the potential of integrating temporal features into LLMs, paving the way for advanced time series analysis.
  arXiv  Detail & Related papers  (2024-10-24T12:32:19Z)
• CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning [59.88924847995279]
  We propose a novel Cross-Modal LLM Fine-Tuning (CALF) framework for MTSF. To reduce the distribution discrepancy, we develop the cross-modal match module. CALF establishes state-of-the-art performance for both long-term and short-term forecasting tasks.
  arXiv  Detail & Related papers  (2024-03-12T04:04:38Z)
• Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning [22.28251586213348]
  aLLM4TS is an innovative framework that adapts Large Language Models (LLMs) for time-series representation learning. A distinctive element of our framework is the patch-wise decoding layer, which departs from previous methods reliant on sequence-level decoding.
  arXiv  Detail & Related papers  (2024-02-07T13:51:26Z)
• AutoTimes: Autoregressive Time Series Forecasters via Large Language Models [67.83502953961505]
  AutoTimes projects time series into the embedding space of language tokens and autoregressively generates future predictions with arbitrary lengths. We formulate time series as prompts, extending the context for prediction beyond the lookback window. AutoTimes achieves state-of-the-art with 0.1% trainable parameters and over $5times$ training/inference speedup.
  arXiv  Detail & Related papers  (2024-02-04T06:59:21Z)
• Large Language Models Are Zero-Shot Time Series Forecasters [48.73953666153385]
  By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. We find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero-shot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks.
  arXiv  Detail & Related papers  (2023-10-11T19:01:28Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)
• TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders [55.00904795497786]
  We propose TimeMAE, a novel self-supervised paradigm for learning transferrable time series representations based on transformer networks. The TimeMAE learns enriched contextual representations of time series with a bidirectional encoding scheme. To solve the discrepancy issue incurred by newly injected masked embeddings, we design a decoupled autoencoder architecture.
  arXiv  Detail & Related papers  (2023-03-01T08:33:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.