Time Series as Language: A Universal Tokenizer for General-Purpose Time Series Foundation Models

• URL: http://arxiv.org/abs/2606.09861v1
• Date: Sun, 31 May 2026 16:04:11 GMT
• Title: Time Series as Language: A Universal Tokenizer for General-Purpose Time Series Foundation Models
• Abstract summary: We introduce UniTok, a universal tokenizer that transforms continuous time series (TS) into discrete tokens, and UniTok-FM, a foundation model pretrained via NTP on these tokens.<n>UniTok-FM is a general-purpose foundation model that supports zero-shot and prompt-boosted forecasting, as well as few-shot generation and classification via training-free in-context inference.
• Score: 58.73838451441261
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Next-Token Prediction (NTP) has unified LLM pretraining, its adaptation to unbounded, continuous time series (TS) remains open. To bridge the gap, we introduce UniTok, a universal tokenizer that transforms TS into discrete tokens, and UniTok-FM, a foundation model pretrained via NTP on these tokens. UniTok-FM is a general-purpose foundation model that supports zero-shot and prompt-boosted forecasting, as well as few-shot generation and classification via training-free in-context inference--a capability not achieved by prior works. Technically, UniTok is a vector-quantized autoencoder incorporating prefix normalization for scale stabilization, a progressive-resolution causal architecture for encoding and decoding, and a structure-preserving reconstruction loss for training. UniTok-FM adopts an off-the-shelf LLM architecture without TS-specific modifications. Instead of pretraining on isolated TS, it performs NTP on context windows formed by multiple series with similar patterns, aiming to capture their shared dynamics. Experiments on forecasting, generation, and classification show that a single unified UniTok-FM consistently outperforms statistical and supervised baselines, achieves competitive performance with task-specific foundation models, and uniquely enables training-free in-context inference across tasks.

Related papers

• TimEE: End-to-end Time Series Classification via In-Context Learning [42.26457535497941]
  Time series classification (TSC) is dominated by a two-stage paradigm.<n>We introduce TimEE, a foundation model for end-to-end TSC via in-context learning.<n>TimEE is the first purely synthetic-pretrained model to reach state-of-the-art performance on the UCR benchmark.
  arXiv  Detail & Related papers  (2026-07-08T14:58:06Z)
• TS-ICL: A Flexible Time-Indexed Foundation Model for Time Series via In-Context Learning [0.764671395172401]
  Current approaches primarily focus on forecasting, while real-world time series are often irregularly and partially observed.<n>We introduce TS-ICL, a novel probabilistic In-Context Learning encoder--regressor Transformer that unifies forecasting and imputation.
  arXiv  Detail & Related papers  (2026-06-04T08:52:21Z)
• Reasoning-Aware Training for Time Series Forecasting [44.62309170542093]
  Time Series Foundation Models (TSFMs) excel at numerical forecasting but operate as black boxes lacking qualitative reasoning.<n>Applying text tokenizers directly to temporal data introduces a modality gap: text tokenizers fragment continuous numerical values, degrading mathematical relationships and exploding sequence lengths.<n>We introduce STRIDE (Strategic Time-series Reasoning Injected via Distilled Embeddings)<n>Instead of discrete tokens, STRIDE distills reasoning traces into a lightweight LLM, dynamically projecting its mean-pooled hidden states as a cross-modal prior into the target numerical encoder.
  arXiv  Detail & Related papers  (2026-05-09T02:41:47Z)
• Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models [102.20309135516186]
  Cross-entropy (CE) training provides dense and scalable supervision for language models.<n>We introduce a feature-matching objective for language-model fine-tuning.<n>To optimize this objective efficiently, we propose energy-based fine-tuning.
  arXiv  Detail & Related papers  (2026-03-12T17:57:50Z)
• UniDiff: A Unified Diffusion Framework for Multimodal Time Series Forecasting [90.47915032778366]
  We propose UniDiff, a unified diffusion framework for multimodal time series forecasting.<n>At its core lies a unified and parallel fusion module, where a single cross-attention mechanism integrates structural information from timestamps and semantic context from texts.<n>Experiments on real-world benchmark datasets across eight domains demonstrate that the proposed UniDiff model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-08T05:36:14Z)
• Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Models [48.61795272482598]
  Vision-Language Models (VLMs) demonstrate impressive zero-shot generalization through large-scale image-text pretraining.<n>But their performance can drop once the deployment distribution diverges from the training distribution.<n>Test-Time Adaptation (TTA) methods update models using unlabeled target data.<n>We propose textbfClass-Aware textbfPrototype textbfL with textbfNegative textbfContrast(textbfCPL-NC), a lightweight TTA framework
  arXiv  Detail & Related papers  (2025-10-22T17:38:35Z)
• UniCast: A Unified Multimodal Prompting Framework for Time Series Forecasting [9.836278124939453]
  Time series forecasting is a foundational task across domains, such as finance, healthcare, and environmental monitoring.<n>Existing models operate predominantly in a unimodal setting, ignoring the rich multimodal context, such as visual and textual signals, that often accompanies time series data in real-world scenarios.<n>This paper introduces a novel parameter-efficient multimodal framework, UniCast, that extends TSFMs to jointly leverage time series, vision, text modalities for enhanced forecasting performance.
  arXiv  Detail & Related papers  (2025-08-16T07:33:27Z)
• UniTS: A Unified Multi-Task Time Series Model [31.675845788410246]
  UniTS is a unified multi-task time series model that integrates predictive and generative tasks into a single framework. UniTS is tested on 38 datasets across human activity sensors, healthcare, engineering, and finance.
  arXiv  Detail & Related papers  (2024-02-29T21:25:58Z)
• Unified Training of Universal Time Series Forecasting Transformers [104.56318980466742]
  We present a Masked-based Universal Time Series Forecasting Transformer (Moirai) Moirai is trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains. Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models.
  arXiv  Detail & Related papers  (2024-02-04T20:00:45Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)
• Structural Self-Supervised Objectives for Transformers [3.018656336329545]
  This thesis focuses on improving the pre-training of natural language models using unsupervised raw data. In the first part, we introduce three alternative pre-training objectives to BERT's Masked Language Modeling (MLM) In the second part, we proposes self-supervised pre-training tasks that align structurally with downstream applications.
  arXiv  Detail & Related papers  (2023-09-15T09:30:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.