MM-ISTS: Cooperating Irregularly Sampled Time Series Forecasting with Multimodal Vision-Text LLMs

• URL: http://arxiv.org/abs/2603.05997v1
• Date: Fri, 06 Mar 2026 07:57:15 GMT
• Title: MM-ISTS: Cooperating Irregularly Sampled Time Series Forecasting with Multimodal Vision-Text LLMs
• Abstract summary: Irregularly sampled time series (ISTS) are widespread in real-world scenarios, exhibiting asynchronous observations on uneven time intervals across variables.<n>We propose MM-ISTS, a framework that bridges temporal, visual, and textual modalities, facilitating ISTS forecasting.<n>In particular, a cross-modal vision-text encoding module is proposed to automatically generate informative visual images and textual data.<n>In parallel, ISTS encoding extracts complementary yet enriched temporal features from historical ISTS observations, including multi-view embedding fusion and a temporal-variable encoder.
• Score: 17.348428689125633
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Irregularly sampled time series (ISTS) are widespread in real-world scenarios, exhibiting asynchronous observations on uneven time intervals across variables. Existing ISTS forecasting methods often solely utilize historical observations to predict future ones while falling short in learning contextual semantics and fine-grained temporal patterns. To address these problems, we achieve MM-ISTS, a multimodal framework augmented by vision-text large language models, that bridges temporal, visual, and textual modalities, facilitating ISTS forecasting. MM-ISTS encompasses a novel two-stage encoding mechanism. In particular, a cross-modal vision-text encoding module is proposed to automatically generate informative visual images and textual data, enabling the capture of intricate temporal patterns and comprehensive contextual understanding, in collaboration with multimodal LLMs (MLLMs). In parallel, ISTS encoding extracts complementary yet enriched temporal features from historical ISTS observations, including multi-view embedding fusion and a temporal-variable encoder. Further, we propose an adaptive query-based feature extractor to compress the learned tokens of MLLMs, filtering out small-scale useful knowledge, which in turn reduces computational costs. In addition, a multimodal alignment module with modality-aware gating is designed to alleviate the modality gap across ISTS, images, and text. Extensive experiments on real data offer insight into the effectiveness of the proposed solutions.

Related papers

• FiCoTS: Fine-to-Coarse LLM-Enhanced Hierarchical Cross-Modality Interaction for Time Series Forecasting [13.70466880923202]
  Time series forecasting is central to data analysis and web technologies.<n>Large Language Models (LLMs) offers significant potential for this field.<n>We propose FiCoTS, an LLM-enhanced fine-to-coarse framework for multimodal time series forecasting.
  arXiv  Detail & Related papers  (2025-11-29T03:17:26Z)
• Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives [22.10401153489018]
  Time series forecasting traditionally relies on unimodal numerical inputs.<n>We propose a multimodal contrastive learning framework that transforms raw time series into structured visual and textual perspectives.
  arXiv  Detail & Related papers  (2025-06-30T17:59:14Z)
• Time-Prompt: Integrated Heterogeneous Prompts for Unlocking LLMs in Time Series Forecasting [13.283980715705693]
  Time series forecasting aims to model temporal dependencies among variables for future state inference.<n>Deep learning-based methods have achieved remarkable progress, but they still exhibit suboptimal performance in long-term forecasting.<n>We propose Time-Prompt, a framework for activating large language models for time series forecasting.
  arXiv  Detail & Related papers  (2025-06-21T08:22:25Z)
• Context-Aware Probabilistic Modeling with LLM for Multimodal Time Series Forecasting [24.56167831047955]
  We propose CAPTime, a context-aware probabilistic multimodal time series forecasting method.<n>Our method first encodes temporal patterns using a pretrained time series encoder, then aligns them with textual contexts via learnable interactions.<n> Experiments on diverse time series forecasting tasks demonstrate the superior accuracy and generalization of CAPTime.
  arXiv  Detail & Related papers  (2025-05-16T01:23:53Z)
• LLM-PS: Empowering Large Language Models for Time Series Forecasting with Temporal Patterns and Semantics [56.99021951927683]
  Time Series Forecasting (TSF) is critical in many real-world domains like financial planning and health monitoring.<n>Existing Large Language Models (LLMs) usually perform suboptimally because they neglect the inherent characteristics of time series data.<n>We propose LLM-PS to empower the LLM for TSF by learning the fundamental textitPatterns and meaningful textitSemantics from time series data.
  arXiv  Detail & Related papers  (2025-03-12T11:45:11Z)
• TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop [79.5773512667468]
  TimeXL is a multi-modal prediction framework that integrates a prototype-based time series encoder with three collaborating Large Language Models.<n>A reflection LLM compares the predicted values against the ground truth, identifying textual inconsistencies or noise.<n>This closed-loop workflow-prediction, critique (reflect), and refinement-continuously boosts the framework's performance and interpretability.
  arXiv  Detail & Related papers  (2025-03-02T20:40:53Z)
• Towards Text-Image Interleaved Retrieval [49.96332254241075]
  We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences.<n>We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.<n>We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
  arXiv  Detail & Related papers  (2025-02-18T12:00:47Z)
• TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agents [52.13094810313054]
  TimeCAP is a time-series processing framework that creatively employs Large Language Models (LLMs) as contextualizers of time series data.<n>TimeCAP incorporates two independent LLM agents: one generates a textual summary capturing the context of the time series, while the other uses this enriched summary to make more informed predictions.<n> Experimental results on real-world datasets demonstrate that TimeCAP outperforms state-of-the-art methods for time series event prediction.
  arXiv  Detail & Related papers  (2025-02-17T04:17:27Z)
• Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting [26.4608782425897]
  Time-VLM is a novel framework that bridges temporal, visual, and textual modalities for enhanced forecasting.<n>Our framework comprises three key components: (1) a Retrieval-Augmented Learner, which extracts enriched temporal features through memory bank interactions; (2) a Vision-Augmented Learner, which encodes time series as informative images; and (3) a Text-Augmented Learner, which generates contextual textual descriptions.
  arXiv  Detail & Related papers  (2025-02-06T05:59:45Z)
• Hierarchical Multimodal LLMs with Semantic Space Alignment for Enhanced Time Series Classification [4.5939667818289385]
  HiTime is a hierarchical multi-modal model that seamlessly integrates temporal information into large language models. Our findings highlight the potential of integrating temporal features into LLMs, paving the way for advanced time series analysis.
  arXiv  Detail & Related papers  (2024-10-24T12:32:19Z)
• RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training [55.54020926284334]
  Multimodal Large Language Models (MLLMs) have recently received substantial interest, which shows their emerging potential as general-purpose models for various vision-language tasks. Retrieval augmentation techniques have proven to be effective plugins for both LLMs and MLLMs. In this study, we propose multimodal adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training (RA-BLIP), a novel retrieval-augmented framework for various MLLMs.
  arXiv  Detail & Related papers  (2024-10-18T03:45:19Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.