Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions

• URL: http://arxiv.org/abs/2505.14543v1
• Date: Tue, 20 May 2025 15:58:54 GMT
• Title: Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions
• Authors: Utsav Dutta, Sina Khoshfetrat Pakazad, Henrik Ohlsson,
• Abstract summary: Traditional time series models are task-specific and often depend on dataset-specific training and extensive feature engineering.<n>We introduce $textbfCHARM$, a foundation embedding model for multivariate time series that learns shared, transferable, and domain-aware representations.<n>The model is trained using a Joint Embedding Predictive Architecture (JEPA), with novel augmentation schemes and a loss function designed to improve interpretability and training stability.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Traditional time series models are task-specific and often depend on dataset-specific training and extensive feature engineering. While Transformer-based architectures have improved scalability, foundation models, commonplace in text, vision, and audio, remain under-explored for time series and are largely restricted to forecasting. We introduce $\textbf{CHARM}$, a foundation embedding model for multivariate time series that learns shared, transferable, and domain-aware representations. To address the unique difficulties of time series foundation learning, $\textbf{CHARM}$ incorporates architectural innovations that integrate channel-level textual descriptions while remaining invariant to channel order. The model is trained using a Joint Embedding Predictive Architecture (JEPA), with novel augmentation schemes and a loss function designed to improve interpretability and training stability. Our $7$M-parameter model achieves state-of-the-art performance across diverse downstream tasks, setting a new benchmark for time series representation learning.

Related papers

• Time Tracker: Mixture-of-Experts-Enhanced Foundation Time Series Forecasting Model with Decoupled Training Pipelines [5.543238821368548]
  Time series often exhibit significant diversity in their temporal patterns across different time spans and domains.<n>Time Tracker achieves state-of-the-art performance in predicting accuracy, model generalization and adaptability.
  arXiv  Detail & Related papers  (2025-05-21T06:18:41Z)
• UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines [64.84631333071728]
  We introduce bfUnistage, a unified Transformer-based framework fortemporal modeling.<n>Our work demonstrates that a task-specific vision-text can build a generalizable model fortemporal learning.<n>We also introduce a temporal module to incorporate temporal dynamics explicitly.
  arXiv  Detail & Related papers  (2025-03-26T17:33:23Z)
• TimesBERT: A BERT-Style Foundation Model for Time Series Understanding [72.64824086839631]
  GPT-style models have been positioned as foundation models for time series forecasting.<n>BERT-style architecture has not been fully unlocked for time series understanding.<n>We design TimesBERT to learn generic representations of time series.<n>Our model is pre-trained on 260 billion time points across diverse domains.
  arXiv  Detail & Related papers  (2025-02-28T17:14:44Z)
• Towards Generalisable Time Series Understanding Across Domains [10.350643783811174]
  We introduce a novel pre-training paradigm specifically designed to handle time series heterogeneity.<n>We propose a tokeniser with learnable domain signatures, a dual masking strategy, and a normalised cross-correlation loss.<n>Our code and pre-trained weights are available at https://www.oetu.com/oetu/otis.
  arXiv  Detail & Related papers  (2024-10-09T17:09:30Z)
• Towards Long-Context Time Series Foundation Models [17.224575072056627]
  Time series foundation models have shown impressive performance on a variety of tasks, across a wide range of domains, even in zero-shot settings. This study bridges the gap by systematically comparing various context expansion techniques from both language and time series domains.
  arXiv  Detail & Related papers  (2024-09-20T14:19:59Z)
• PLUTUS: A Well Pre-trained Large Unified Transformer can Unveil Financial Time Series Regularities [0.848210898747543]
  Financial time series modeling is crucial for understanding and predicting market behaviors. Traditional models struggle to capture complex patterns due to non-linearity, non-stationarity, and high noise levels. Inspired by the success of large language models in NLP, we introduce $textbfPLUTUS$, a $textbfP$re-trained $textbfL$arge. PLUTUS is the first open-source, large-scale, pre-trained financial time series model with over one billion parameters.
  arXiv  Detail & Related papers  (2024-08-19T15:59:46Z)
• Deep Time Series Models: A Comprehensive Survey and Benchmark [74.28364194333447]
  Time series data is of great significance in real-world scenarios. Recent years have witnessed remarkable breakthroughs in the time series community. We release Time Series Library (TSLib) as a fair benchmark of deep time series models for diverse analysis tasks.
  arXiv  Detail & Related papers  (2024-07-18T08:31:55Z)
• Unified Training of Universal Time Series Forecasting Transformers [104.56318980466742]
  We present a Masked-based Universal Time Series Forecasting Transformer (Moirai) Moirai is trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains. Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models.
  arXiv  Detail & Related papers  (2024-02-04T20:00:45Z)
• Timer: Generative Pre-trained Transformers Are Large Time Series Models [83.03091523806668]
  This paper aims at the early development of large time series models (LTSM) During pre-training, we curate large-scale datasets with up to 1 billion time points. To meet diverse application needs, we convert forecasting, imputation, and anomaly detection of time series into a unified generative task.
  arXiv  Detail & Related papers  (2024-02-04T06:55:55Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.