Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data

• URL: http://arxiv.org/abs/2411.06735v2
• Date: Thu, 21 Nov 2024 00:52:53 GMT
• Title: Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data
• Authors: Kai Kim, Howard Tsai, Rajat Sen, Abhimanyu Das, Zihao Zhou, Abhishek Tanpure, Mathew Luo, Rose Yu,
• Abstract summary: Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series. We develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare.
• Score: 23.10730301634422
• License:
• Abstract: Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting.

Related papers

• Tackling Data Heterogeneity in Federated Time Series Forecasting [61.021413959988216]
  Time series forecasting plays a critical role in various real-world applications, including energy consumption prediction, disease transmission monitoring, and weather forecasting. Most existing methods rely on a centralized training paradigm, where large amounts of data are collected from distributed devices to a central cloud server. We propose a novel framework, Fed-TREND, to address data heterogeneity by generating informative synthetic data as auxiliary knowledge carriers.
  arXiv  Detail & Related papers  (2024-11-24T04:56:45Z)
• Text2Freq: Learning Series Patterns from Text via Frequency Domain [8.922661807801227]
  Text2Freq is a cross-modality model that integrates text and time series data via the frequency domain. Our experiments on paired datasets of real-world stock prices and synthetic texts show that Text2Freq achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-11-01T16:11:02Z)
• Hierarchical Multimodal LLMs with Semantic Space Alignment for Enhanced Time Series Classification [4.5939667818289385]
  HiTime is a hierarchical multi-modal model that seamlessly integrates temporal information into large language models. Our findings highlight the potential of integrating temporal features into LLMs, paving the way for advanced time series analysis.
  arXiv  Detail & Related papers  (2024-10-24T12:32:19Z)
• TimeAutoDiff: Combining Autoencoder and Diffusion model for time series tabular data synthesizing [13.385264002435145]
  In this paper, we leverage the power of latent diffusion models to generate synthetic time series tabular data. We tackle this problem by combining the ideas of the variational auto-encoder (VAE) and the denoising diffusion probabilistic model (DDPM) Our model named as textttTimeAutoDiff has several key advantages including (1) Generality: the ability to handle the broad spectrum of time series data from single to multi-sequence datasets.
  arXiv  Detail & Related papers  (2024-06-23T06:32:27Z)
• Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis [40.44013652777716]
  Time-MMD is the first multi-domain, multimodal time series dataset. MM-TSFlib is the first multimodal time-series forecasting library.
  arXiv  Detail & Related papers  (2024-06-12T20:20:09Z)
• Large Language Models Are Zero-Shot Time Series Forecasters [48.73953666153385]
  By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. We find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero-shot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks.
  arXiv  Detail & Related papers  (2023-10-11T19:01:28Z)
• Align and Attend: Multimodal Summarization with Dual Contrastive Losses [57.83012574678091]
  The goal of multimodal summarization is to extract the most important information from different modalities to form output summaries. Existing methods fail to leverage the temporal correspondence between different modalities and ignore the intrinsic correlation between different samples. We introduce Align and Attend Multimodal Summarization (A2Summ), a unified multimodal transformer-based model which can effectively align and attend the multimodal input.
  arXiv  Detail & Related papers  (2023-03-13T17:01:42Z)
• Multi-scale Attention Flow for Probabilistic Time Series Forecasting [68.20798558048678]
  We propose a novel non-autoregressive deep learning model, called Multi-scale Attention Normalizing Flow(MANF) Our model avoids the influence of cumulative error and does not increase the time complexity. Our model achieves state-of-the-art performance on many popular multivariate datasets.
  arXiv  Detail & Related papers  (2022-05-16T07:53:42Z)
• Benchmarking Multimodal AutoML for Tabular Data with Text Fields [83.43249184357053]
  We assemble 18 multimodal data tables that each contain some text fields. Our benchmark enables researchers to evaluate their own methods for supervised learning with numeric, categorical, and text features.
  arXiv  Detail & Related papers  (2021-11-04T09:29:16Z)
• Instance-wise Graph-based Framework for Multivariate Time Series Forecasting [69.38716332931986]
  We propose a simple yet efficient instance-wise graph-based framework to utilize the inter-dependencies of different variables at different time stamps. The key idea of our framework is aggregating information from the historical time series of different variables to the current time series that we need to forecast.
  arXiv  Detail & Related papers  (2021-09-14T07:38:35Z)
• A Deep Structural Model for Analyzing Correlated Multivariate Time Series [11.009809732645888]
  We present a deep learning structural time series model which can handle correlated multivariate time series input. The model explicitly learns/extracts the trend, seasonality, and event components. We compare our model with several state-of-the-art methods through a comprehensive set of experiments on a variety of time series data sets.
  arXiv  Detail & Related papers  (2020-01-02T18:48:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.