Does Text Actually Help? Uncovering and Resolving Text Collapse in Multimodal Time Series Forecasting

• URL: http://arxiv.org/abs/2606.19413v1
• Date: Wed, 17 Jun 2026 15:47:26 GMT
• Title: Does Text Actually Help? Uncovering and Resolving Text Collapse in Multimodal Time Series Forecasting
• Abstract summary: We argue that text collapse is a consequence of a fundamental asymmetry in time series forecasting.<n>We propose textbfREST-TS (textbfResidual-textbfExclusive textbfSupervision for textbfText in textbfTime textbfSeries), which turns the asymmetry into a design principle.
• Score: 7.885089945243776
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal time series forecasting, which pairs numerical sequences with domain-relevant textual reports, promises to inject world knowledge into forecasting pipelines. However, we uncover a critical failure mode in existing frameworks that we term text collapse: the text branch converges to a content-independent transformation, contributing negligible discriminative signal regardless of the input description. We argue that text collapse is a consequence of a fundamental asymmetry in time series forecasting: the numerical input is strongly autocorrelated with the output, making the numerical backbone inherently dominant, while the text branch, despite carrying complementary and often critical information, is insufficiently utilized, leading to its systematic underexploitation. To address this, we propose \textbf{REST-TS} (\textbf{R}esidual-\textbf{E}xclusive \textbf{S}upervision for \textbf{T}ext in \textbf{T}ime \textbf{S}eries), which turns the asymmetry into a design principle: the numerical backbone produces its own independent numerical forecast, and the text branch is exclusively supervised to predict the structured components of the residual, the prediction gap that numbers cannot explain. Because no numerical pathway can reduce these losses, the text branch must extract genuine content from the input description. Evaluated across diverse real-world domains and backbone architectures, REST-TS achieves state-of-the-art performance and consistently demonstrates greater text-branch utilization than existing frameworks, providing strong empirical evidence that supervising the text branch on the residual compels it to extract genuine content from the input.

Related papers

• Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents [16.82888862043366]
  We propose a Kullback-Leibler-guided framework for pruning accumulated reasoning in tool-use agents.<n> SPARE achieves the highest average accuracy among pruning methods while removing 37.89-64.58% of reasoning tokens.<n>This favorable accuracy-context trade-off shows that reducing textual dominance restores reliance on visual evidence and mitigates over-conditioning on self-generated language.
  arXiv  Detail & Related papers  (2026-08-24T08:29:17Z)
• Semantics or Structure? Auditing Text Sensitivity in Multimodal Time-Series Forecasting [7.094715131203089]
  Multimodal time-series forecasting has emerged as a promising paradigm in which natural-language context is expected to improve predictive performance.<n>Recent multimodal foundation models, including Aurora, report substantial gains over unimodal baselines on the Time-MMD benchmark, attributing these improvements to textual information.<n>We address this question through controlled text perturbations, attribution analyses, and probes of Aurora's text pathway.<n>We conclude that, on this benchmark and within this family of frozen-encoder architectures, text content is not the operative signal behind the reported gains.
  arXiv  Detail & Related papers  (2026-08-23T09:35:54Z)
• PURA: Provably Unbiased and Robust Multi-Bit Watermarking for AI-Generated Text Attribution [83.66032773976904]
  We present PURA, a provably unbiased and robust multi-bit watermarking method for text attribution.<n>Instead of perturbing token probabilities directly, PURA embeds payloads in the latent sampling space via keyed inverse transform sampling.<n>We show that PURA substantially outperforms existing unbiased baselines in the high-payload regime.
  arXiv  Detail & Related papers  (2026-08-23T04:44:27Z)
• Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics [49.443264461057645]
  Diffusion and continuous flow-based language models have emerged as the leading non-autoregressive alternatives to language modeling.<n>Progress in both paradigms is overwhelmingly tracked by generative perplexity (gen-PPL): the per-token negative log-likelihood of samples under a frozen autoregressive (AR) scorer.<n>We argue that this metric is unsound. By construction, gen-PPL measures only predictability under the scoring AR, not grammaticality or semantic coherence.
  arXiv  Detail & Related papers  (2026-06-07T02:35:56Z)
• PRISM: Prior Rectification and Uncertainty-Aware Structure Modeling for Diffusion-Based Text Image Super-Resolution [51.96078493242164]
  PRISM is a single-step diffusion-based Text-SR framework.<n> PRISM achieves state-of-the-art performance with millisecond-level inference.
  arXiv  Detail & Related papers  (2026-05-13T05:31:06Z)
• Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck [60.930825621351794]
  We introduce Conditional Multimodal Information Bottleneck (CMIB), a method for multimodal skill construction.<n>Unlike naive two-stream formulations, CMIB explicitly conditions the multimodal latent on the text skill, thus structurally reducing cross-modal redundancy.<n>We instantiate CMIB with a variational objective that makes its conditional decomposition tractable to optimize, yielding reusable multimodal skills.
  arXiv  Detail & Related papers  (2026-05-08T22:17:54Z)
• Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings [30.943998879066857]
  Mean pooling, which averages token embeddings, is the standard approach to constructing text embeddings.<n>This paper examines whether mean pooling actually works well in real models.
  arXiv  Detail & Related papers  (2026-04-30T04:09:26Z)
• From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space [30.02774781484044]
  We show that existing methods over-attend to redundant tokens and struggle to reliably translate textual semantics into usable numerical cues.<n>We propose TESS, which introduces a Temporal Evolution Semantic Space as an intermediate bottleneck between modalities.<n> Experiments on four real-world datasets demonstrate up to a 29 percent reduction in forecasting error compared to state-of-the-art unimodal and multimodal baselines.
  arXiv  Detail & Related papers  (2026-03-13T05:11:54Z)
• A Context-Driven Training-Free Network for Lightweight Scene Text Segmentation and Recognition [32.142713322062306]
  Text recognition systems often depend on large end-to-end architectures that require extensive training and are prohibitively expensive for real-time scenarios.<n>We propose a training-free plug-and-play framework that leverages the strengths of pre-trained text recognizers while minimizing redundant computations.<n>Our approach uses context-based understanding and introduces an attention-based segmentation stage, which refines candidate text regions at the pixel level.
  arXiv  Detail & Related papers  (2025-03-19T18:51:01Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Interpretable Automatic Fine-grained Inconsistency Detection in Text Summarization [56.94741578760294]
  We propose the task of fine-grained inconsistency detection, the goal of which is to predict the fine-grained types of factual errors in a summary. Motivated by how humans inspect factual inconsistency in summaries, we propose an interpretable fine-grained inconsistency detection model, FineGrainFact.
  arXiv  Detail & Related papers  (2023-05-23T22:11:47Z)
• TPS++: Attention-Enhanced Thin-Plate Spline for Scene Text Recognition [78.67283660198403]
  Text irregularities pose significant challenges to scene text recognizers. TPS++ is an attention-enhanced TPS transformation that incorporates the attention mechanism to text rectification. It consistently improves the recognition and achieves state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2023-05-09T10:16:43Z)
• BOTD: Bold Outline Text Detector [85.33700624095181]
  We propose a new one-stage text detector, termed as Bold Outline Text Detector (BOTD) BOTD is able to process the arbitrary-shaped text with low model complexity. Experimental results on three real-world benchmarks show the state-of-the-art performance of BOTD.
  arXiv  Detail & Related papers  (2020-11-30T11:54:14Z)
• Text Perceptron: Towards End-to-End Arbitrary-Shaped Text Spotting [49.768327669098674]
  We propose an end-to-end trainable text spotting approach named Text Perceptron. It first employs an efficient segmentation-based text detector that learns the latent text reading order and boundary information. Then a novel Shape Transform Module (abbr. STM) is designed to transform the detected feature regions into regular morphologies.
  arXiv  Detail & Related papers  (2020-02-17T08:07:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.