Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data

• URL: http://arxiv.org/abs/2411.06735v2
• Date: Thu, 21 Nov 2024 00:52:53 GMT
• Title: Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data
• Authors: Kai Kim, Howard Tsai, Rajat Sen, Abhimanyu Das, Zihao Zhou, Abhishek Tanpure, Mathew Luo, Rose Yu,
• Abstract summary: Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series. We develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare.
• Score: 23.10730301634422
• License:
• Abstract: Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting.

Related papers

• TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agents [52.13094810313054]
  TimeCAP is a time-series processing framework that creatively employs Large Language Models (LLMs) as contextualizers of time series data. TimeCAP incorporates two independent LLM agents: one generates a textual summary capturing the context of the time series, while the other uses this enriched summary to make more informed predictions. Experimental results on real-world datasets demonstrate that TimeCAP outperforms state-of-the-art methods for time series event prediction.
  arXiv  Detail & Related papers  (2025-02-17T04:17:27Z)
• Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrative [65.84249211767921]
  Texts as Time Series (TaTS) considers the time-series-paired texts to be auxiliary variables of the time series. TaTS can be plugged into any existing numerical-only time series models and enable them to handle time series data with paired texts effectively.
  arXiv  Detail & Related papers  (2025-02-13T03:43:27Z)
• Unveiling the Potential of Text in High-Dimensional Time Series Forecasting [12.707274099874384]
  We propose a novel framework that integrates time series models with Large Language Models. Inspired by multimodal models, our method combines time series and textual data in the dual-tower structure. Experiments demonstrate that incorporating text enhances high-dimensional time series forecasting performance.
  arXiv  Detail & Related papers  (2025-01-13T04:10:45Z)
• ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data [26.300515935897415]
  ChatTime is a unified framework for time series and text processing. As an out-of-the-box multimodal time series foundation model, ChatTime provides zero-shot forecasting capability. We design a series of experiments to verify the superior performance of ChatTime across multiple tasks and scenarios.
  arXiv  Detail & Related papers  (2024-12-16T02:04:06Z)
• TimeAutoDiff: Combining Autoencoder and Diffusion model for time series tabular data synthesizing [13.385264002435145]
  In this paper, we leverage the power of latent diffusion models to generate synthetic time series tabular data. We tackle this problem by combining the ideas of the variational auto-encoder (VAE) and the denoising diffusion probabilistic model (DDPM) Our model named as textttTimeAutoDiff has several key advantages including (1) Generality: the ability to handle the broad spectrum of time series data from single to multi-sequence datasets.
  arXiv  Detail & Related papers  (2024-06-23T06:32:27Z)
• Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis [40.44013652777716]
  Time-MMD is the first multi-domain, multimodal time series dataset. MM-TSFlib is the first-cut multimodal time-series forecasting library.
  arXiv  Detail & Related papers  (2024-06-12T20:20:09Z)
• TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment [21.690191536424567]
  TimeCMA is an intuitive yet effective framework for time series forecasting. Extensive experiments on eight real datasets demonstrate that TimeCMA outperforms state-of-the-arts.
  arXiv  Detail & Related papers  (2024-06-03T00:27:29Z)
• Large Language Models Are Zero-Shot Time Series Forecasters [48.73953666153385]
  By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. We find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero-shot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks.
  arXiv  Detail & Related papers  (2023-10-11T19:01:28Z)
• Align and Attend: Multimodal Summarization with Dual Contrastive Losses [57.83012574678091]
  The goal of multimodal summarization is to extract the most important information from different modalities to form output summaries. Existing methods fail to leverage the temporal correspondence between different modalities and ignore the intrinsic correlation between different samples. We introduce Align and Attend Multimodal Summarization (A2Summ), a unified multimodal transformer-based model which can effectively align and attend the multimodal input.
  arXiv  Detail & Related papers  (2023-03-13T17:01:42Z)
• Multi-scale Attention Flow for Probabilistic Time Series Forecasting [68.20798558048678]
  We propose a novel non-autoregressive deep learning model, called Multi-scale Attention Normalizing Flow(MANF) Our model avoids the influence of cumulative error and does not increase the time complexity. Our model achieves state-of-the-art performance on many popular multivariate datasets.
  arXiv  Detail & Related papers  (2022-05-16T07:53:42Z)
• Instance-wise Graph-based Framework for Multivariate Time Series Forecasting [69.38716332931986]
  We propose a simple yet efficient instance-wise graph-based framework to utilize the inter-dependencies of different variables at different time stamps. The key idea of our framework is aggregating information from the historical time series of different variables to the current time series that we need to forecast.
  arXiv  Detail & Related papers  (2021-09-14T07:38:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.