From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space

• URL: http://arxiv.org/abs/2603.12664v2
• Date: Mon, 16 Mar 2026 10:24:41 GMT
• Title: From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space
• Abstract summary: We show that existing methods over-attend to redundant tokens and struggle to reliably translate textual semantics into usable numerical cues.<n>We propose TESS, which introduces a Temporal Evolution Semantic Space as an intermediate bottleneck between modalities.<n> Experiments on four real-world datasets demonstrate up to a 29 percent reduction in forecasting error compared to state-of-the-art unimodal and multimodal baselines.
• Score: 30.02774781484044
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Incorporating textual information into time-series forecasting holds promise for addressing event-driven non-stationarity; however, a fundamental modality gap hinders effective fusion: textual descriptions express temporal impacts implicitly and qualitatively, whereas forecasting models rely on explicit and quantitative signals. Through controlled semi-synthetic experiments, we show that existing methods over-attend to redundant tokens and struggle to reliably translate textual semantics into usable numerical cues. To bridge this gap, we propose TESS, which introduces a Temporal Evolution Semantic Space as an intermediate bottleneck between modalities. This space consists of interpretable, numerically grounded temporal primitives (mean shift, volatility, shape, and lag) extracted from text by an LLM via structured prompting and filtered through confidence-aware gating. Experiments on four real-world datasets demonstrate up to a 29 percent reduction in forecasting error compared to state-of-the-art unimodal and multimodal baselines. The code will be released after acceptance.

Related papers

• Semantics or Structure? Auditing Text Sensitivity in Multimodal Time-Series Forecasting [7.094715131203089]
  Multimodal time-series forecasting has emerged as a promising paradigm in which natural-language context is expected to improve predictive performance.<n>Recent multimodal foundation models, including Aurora, report substantial gains over unimodal baselines on the Time-MMD benchmark, attributing these improvements to textual information.<n>We address this question through controlled text perturbations, attribution analyses, and probes of Aurora's text pathway.<n>We conclude that, on this benchmark and within this family of frozen-encoder architectures, text content is not the operative signal behind the reported gains.
  arXiv  Detail & Related papers  (2026-08-23T09:35:54Z)
• From Prompts to Pavement Through Time: Temporal Grounding in Agentic Scene-to-Plan Reasoning [1.2777067998526854]
  This work explores whether temporal conditioning within inter-agent communication can preserve or enhance coherence without introducing degradation in semantic or logical consistency.<n>Results show that while temporal conditioning reshapes reasoning style, it yields no statistically significant improvements in standard NLP-based correctness metrics.<n>These findings clarify the limits of prompt-based temporal grounding and establish the first empirical benchmark for temporal scene-to-plan reasoning.
  arXiv  Detail & Related papers  (2026-05-19T13:18:35Z)
• TimeSAF: Towards LLM-Guided Semantic Asynchronous Fusion for Time Series Forecasting [6.1337977581640075]
  TimeSAF is a new framework based on hierarchical asynchronous fusion.<n>It provides stable and efficient semantic guidance while avoiding interference with low-level temporal dynamics.
  arXiv  Detail & Related papers  (2026-04-14T12:18:00Z)
• Overcoming the Modality Gap in Context-Aided Forecasting [54.976964834365056]
  We introduce a semi-synthetic data augmentation method that generates contexts both descriptive of temporal dynamics and verifiably complementary to numerical histories.<n>This approach enables massive-scale dataset creation, resulting in CAF-7M, a corpus of 7 million context-augmented time series windows.
  arXiv  Detail & Related papers  (2026-03-12T21:05:33Z)
• Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Models [11.196851704643406]
  Diffusion language models (Diffusion-LMs) introduce an explicit temporal dimension into text generation.<n>We show that Diffusion-LMs, like diffusion models in image generation, exhibit a temporal division of labor.<n>We propose Time-Annealed Perturbation Sampling (TAPS), a training-free inference strategy that encourages semantic branching early in the diffusion process.<n>TAPS is compatible with both non-autoregressive and semi-autoregressive Diffusion backbones, demonstrated on LLaDA and TraDo in our paper, and consistently improves output diversity across creative writing and reasoning benchmarks without compromising generation quality.
  arXiv  Detail & Related papers  (2026-01-30T06:39:33Z)
• BiPrompt: Bilateral Prompt Optimization for Visual and Textual Debiasing in Vision-Language Models [7.174865411448373]
  We propose a bilateral prompt optimization framework (BiPrompt) that simultaneously mitigates non-causal feature reliance in both modalities during test-time adaptation.<n>On the visual side, it employs structured attention-guided erasure to suppress background activations and enforce prediction consistency between causal and spurious regions.<n>On the textual side, it introduces balanced prompt normalization, a learnable re-centering mechanism that aligns class embeddings toward an isotropic semantic space.
  arXiv  Detail & Related papers  (2026-01-05T14:22:20Z)
• UniDiff: A Unified Diffusion Framework for Multimodal Time Series Forecasting [90.47915032778366]
  We propose UniDiff, a unified diffusion framework for multimodal time series forecasting.<n>At its core lies a unified and parallel fusion module, where a single cross-attention mechanism integrates structural information from timestamps and semantic context from texts.<n>Experiments on real-world benchmark datasets across eight domains demonstrate that the proposed UniDiff model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-08T05:36:14Z)
• Priors in Time: Missing Inductive Biases for Language Model Interpretability [58.07412640266836]
  We show that Sparse Autoencoders impose priors that assume independence of concepts across time, implying stationarity.<n>We introduce a new interpretability objective -- Temporal Feature Analysis -- which possesses a temporal inductive bias to decompose representations at a given time into two parts.<n>Our results underscore the need for inductive biases that match the data in designing robust interpretability tools.
  arXiv  Detail & Related papers  (2025-11-03T18:43:48Z)
• How Different from the Past? Spatio-Temporal Time Series Forecasting with Self-Supervised Deviation Learning [15.102926671713668]
  We propose ST-SSDL, a Spatio-Temporal series time forecasting framework.<n>It discretizes latent space using learnable prototypes that represent typicaltemporal patterns.<n>Experiments show that ST-SSDL consistently outperforms state-of-the-art baselines across multiple metrics.
  arXiv  Detail & Related papers  (2025-10-06T15:21:13Z)
• Explaining multimodal LLMs via intra-modal token interactions [55.27436637894534]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet their internal decision-making mechanisms remain insufficiently understood.<n>We propose enhancing interpretability by leveraging intra-modal interaction.
  arXiv  Detail & Related papers  (2025-09-26T14:39:13Z)
• EventTSF: Event-Aware Non-Stationary Time Series Forecasting [73.54313384419792]
  EventTSF is an autoregressive generation framework that integrates historical time series with textual events to make subsequent forecasts.<n>Experiments on 8 synthetic and real-world datasets show that EventTSF outperforms 12 baselines across diverse event-aware non-stationary time series forecasting scenarios.
  arXiv  Detail & Related papers  (2025-08-19T01:28:47Z)
• Time-Prompt: Integrated Heterogeneous Prompts for Unlocking LLMs in Time Series Forecasting [13.283980715705693]
  Time series forecasting aims to model temporal dependencies among variables for future state inference.<n>Deep learning-based methods have achieved remarkable progress, but they still exhibit suboptimal performance in long-term forecasting.<n>We propose Time-Prompt, a framework for activating large language models for time series forecasting.
  arXiv  Detail & Related papers  (2025-06-21T08:22:25Z)
• RePST: Language Model Empowered Spatio-Temporal Forecasting via Semantic-Oriented Reprogramming [24.9561009415531]
  We aim to harness the reasoning and generalization abilities of Pre-trained Language Models (PLMs) for intricate-temporal forecasting.<n>We propose RePST, a semantic-oriented PLM reprogramming framework tailored fortemporal forecasting.
  arXiv  Detail & Related papers  (2024-08-24T07:59:36Z)
• Interactive Test-Time Adaptation with Reliable Spatial-Temporal Voxels for Multi-Modal Segmentation [56.70910056845503]
  Multi-modal test-time adaptation (MM-TTA) adapts models to an unlabeled target domain by leveraging the complementary multi-modal inputs in an online manner.<n>Previous MM-TTA methods for 3D segmentation suffer from two major limitations: 1) unstable frame-wise predictions caused by temporal inconsistency, and 2) consistently incorrect predictions that violate the assumption of reliable modality guidance.<n>This work introduces a comprehensive two-fold framework: Latte++ that better suppresses the unstable frame-wise predictions with more informative geometric correspondences, and Interactive Test-Time Adaptation (ITTA), a flexible add-on to empower effortless human feedback
  arXiv  Detail & Related papers  (2024-03-11T06:56:08Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.