Semantics or Structure? Auditing Text Sensitivity in Multimodal Time-Series Forecasting

• URL: http://arxiv.org/abs/2608.22321v1
• Date: Sun, 23 Aug 2026 09:35:54 GMT
• Title: Semantics or Structure? Auditing Text Sensitivity in Multimodal Time-Series Forecasting
• Abstract summary: Multimodal time-series forecasting has emerged as a promising paradigm in which natural-language context is expected to improve predictive performance.<n>Recent multimodal foundation models, including Aurora, report substantial gains over unimodal baselines on the Time-MMD benchmark, attributing these improvements to textual information.<n>We address this question through controlled text perturbations, attribution analyses, and probes of Aurora's text pathway.<n>We conclude that, on this benchmark and within this family of frozen-encoder architectures, text content is not the operative signal behind the reported gains.
• Score: 7.094715131203089
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal time-series forecasting has emerged as a promising paradigm in which natural-language context is expected to improve predictive performance. Recent multimodal foundation models, including Aurora, as well as early- and late-fusion approaches such as MM-TSFlib and TaTS, report substantial gains over unimodal baselines on the Time-MMD benchmark, attributing these improvements to textual information. However, whether these models are actually sensitive to the semantic content of the text remains unverified. We address this question through controlled text perturbations, attribution analyses, and probes of Aurora's text pathway. On Time-MMD, swapping each row's text for any other real text (empty, constant, within-domain shuffled, or cross-domain) moves mean MSE by less than $0.5\%$ on all three architectures. The improvement reported in the literature is recovered when a co-shipped numeric column is removed without touching text. We conclude that, on this benchmark and within this family of frozen-encoder architectures, text content is not the operative signal behind the reported gains. To support future work on text integration in multimodal foundation models for structured data, we release our perturbation protocol and evaluation harness as a reusable diagnostic toolkit.

Related papers

• Does Text Actually Help? Uncovering and Resolving Text Collapse in Multimodal Time Series Forecasting [7.885089945243776]
  We argue that text collapse is a consequence of a fundamental asymmetry in time series forecasting.<n>We propose textbfREST-TS (textbfResidual-textbfExclusive textbfSupervision for textbfText in textbfTime textbfSeries), which turns the asymmetry into a design principle.
  arXiv  Detail & Related papers  (2026-06-17T15:47:26Z)
• Continuous Latent Diffusion Language Model [48.974403879186916]
  Large language models have achieved remarkable success under the autoregressive paradigm.<n>Existing alternatives still struggle to jointly achieve generation efficiency, scalable representation learning, and effective global semantic modeling.<n>We propose Cola DLM, a hierarchical latent diffusion language model that frames text generation through hierarchical information decomposition.
  arXiv  Detail & Related papers  (2026-05-07T16:44:56Z)
• Reward-Guided Semantic Evolution for Test-time Adaptive Object Detection [82.2968697030677]
  Open-vocabulary object detection with vision-language models (VLMs) such as Grounding DINO suffers from performance degradation under test-time distribution shifts.<n>We propose Reward-Guided Semantic Evolution (RGSE), a training-free framework that directly refines the text embeddings at test time.
  arXiv  Detail & Related papers  (2026-05-06T06:17:41Z)
• From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space [30.02774781484044]
  We show that existing methods over-attend to redundant tokens and struggle to reliably translate textual semantics into usable numerical cues.<n>We propose TESS, which introduces a Temporal Evolution Semantic Space as an intermediate bottleneck between modalities.<n> Experiments on four real-world datasets demonstrate up to a 29 percent reduction in forecasting error compared to state-of-the-art unimodal and multimodal baselines.
  arXiv  Detail & Related papers  (2026-03-13T05:11:54Z)
• Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysis [1.7522684436505962]
  This paper proposes the Text-routed sparse mixture-of-Experts model with eXplanation and Temporal alignment for MSA.<n> TEXT first augments explanations for MSA via Multi-modal Large Language Models (MLLM), and then novelly aligns the epresentations of audio and video through a temporality-oriented neural network block.<n> TEXT achieves the best performance cross four datasets among all tested models, including three recently proposed approaches and three MLLMs.
  arXiv  Detail & Related papers  (2025-12-28T01:58:30Z)
• UniDiff: A Unified Diffusion Framework for Multimodal Time Series Forecasting [90.47915032778366]
  We propose UniDiff, a unified diffusion framework for multimodal time series forecasting.<n>At its core lies a unified and parallel fusion module, where a single cross-attention mechanism integrates structural information from timestamps and semantic context from texts.<n>Experiments on real-world benchmark datasets across eight domains demonstrate that the proposed UniDiff model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-08T05:36:14Z)
• BALM-TSF: Balanced Multimodal Alignment for LLM-Based Time Series Forecasting [5.360725360679271]
  BALM-TSF is a lightweight framework for time series forecasting.<n>It maintains balance between time series and textual embeddings.<n>It achieves state-of-the-art performance in both long-term and few-shot forecasting.
  arXiv  Detail & Related papers  (2025-08-30T22:31:55Z)
• BRIDGE: Bootstrapping Text to Control Time-Series Generation via Multi-Agent Iterative Optimization and Diffusion Modeling [51.830134409330704]
  Time-series Generation (TSG) is a prominent research area with broad applications in simulations, data augmentation, and counterfactual analysis.<n>We argue that text can provide semantic insights, domain information and instance-specific temporal patterns, to guide and improve TSG.<n>We introduce BRIDGE, a hybrid text-controlled TSG framework that integrates semantic prototypes with text description for supporting domain-level guidance.
  arXiv  Detail & Related papers  (2025-03-04T09:40:00Z)
• Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrative [65.84249211767921]
  Texts as Time Series (TaTS) can be plugged into any existing numerical-only time series models.<n>We show that TaTS can enhance predictive performance without modifying model architectures.
  arXiv  Detail & Related papers  (2025-02-13T03:43:27Z)
• Discourse Features Enhance Detection of Document-Level Machine-Generated Content [53.41994768824785]
  Machine-generated content poses challenges such as academic plagiarism and the spread of misinformation.<n>Existing MGC detectors often focus solely on surface-level information, overlooking implicit and structural features.<n>We introduce novel methodologies and datasets to overcome these challenges.
  arXiv  Detail & Related papers  (2024-12-17T08:47:41Z)
• TS-HTFA: Advancing Time Series Forecasting via Hierarchical Text-Free Alignment with Large Language Models [14.411646409316624]
  We introduce textbfHierarchical textbfText-textbfFree textbfAlignment (textbfTS-HTFA), a novel method for time-series forecasting.<n>We replace paired text data with adaptive virtual text based on QR decomposition word embeddings and learnable prompt.<n>Experiments on multiple time-series benchmarks demonstrate that HTFA achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-09-23T12:57:24Z)
• MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation [102.20036684996248]
  We propose MURMUR, a neuro-symbolic modular approach to text generation from semi-structured data with multi-step reasoning. We conduct experiments on two data-to-text generation tasks like WebNLG and LogicNLG.
  arXiv  Detail & Related papers  (2022-12-16T17:36:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.