TimesBERT: A BERT-Style Foundation Model for Time Series Understanding

• URL: http://arxiv.org/abs/2502.21245v1
• Date: Fri, 28 Feb 2025 17:14:44 GMT
• Title: TimesBERT: A BERT-Style Foundation Model for Time Series Understanding
• Authors: Haoran Zhang, Yong Liu, Yunzhong Qiu, Haixuan Liu, Zhongyi Pei, Jianmin Wang, Mingsheng Long,
• Abstract summary: GPT-style models have been positioned as foundation models for time series forecasting.<n>BERT-style architecture has not been fully unlocked for time series understanding.<n>We design TimesBERT to learn generic representations of time series.<n>Our model is pre-trained on 260 billion time points across diverse domains.
• Score: 72.64824086839631
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Time series analysis is crucial in diverse scenarios. Beyond forecasting, considerable real-world tasks are categorized into classification, imputation, and anomaly detection, underscoring different capabilities termed time series understanding in this paper. While GPT-style models have been positioned as foundation models for time series forecasting, the BERT-style architecture, which has made significant advances in natural language understanding, has not been fully unlocked for time series understanding, possibly attributed to the undesirable dropout of essential elements of BERT. In this paper, inspired by the shared multi-granularity structure between multivariate time series and multisentence documents, we design TimesBERT to learn generic representations of time series including temporal patterns and variate-centric characteristics. In addition to a natural adaptation of masked modeling, we propose a parallel task of functional token prediction to embody vital multi-granularity structures. Our model is pre-trained on 260 billion time points across diverse domains. Leveraging multi-granularity representations, TimesBERT achieves state-of-the-art performance across four typical downstream understanding tasks, outperforming task-specific models and language pre-trained backbones, positioning it as a versatile foundation model for time series understanding.

Related papers

• Language in the Flow of Time: Time-Series-Paired Texts Weaved into a Unified Temporal Narrative [65.84249211767921]
  Texts as Time Series (TaTS) considers the time-series-paired texts to be auxiliary variables of the time series.<n>TaTS can be plugged into any existing numerical-only time series models and enable them to handle time series data with paired texts effectively.
  arXiv  Detail & Related papers  (2025-02-13T03:43:27Z)
• General Time-series Model for Universal Knowledge Representation of Multivariate Time-Series data [61.163542597764796]
  We show that time series with different time granularities (or corresponding frequency resolutions) exhibit distinct joint distributions in the frequency domain.<n>A novel Fourier knowledge attention mechanism is proposed to enable learning time-aware representations from both the temporal and frequency domains.<n>An autoregressive blank infilling pre-training framework is incorporated to time series analysis for the first time, leading to a generative tasks agnostic pre-training strategy.
  arXiv  Detail & Related papers  (2025-02-05T15:20:04Z)
• TempoGPT: Enhancing Time Series Reasoning via Quantizing Embedding [13.996105878417204]
  We propose a multi-modal time series data construction approach and a multi-modal time series language model (TLM), TempoGPT. We construct multi-modal data for complex reasoning tasks by analyzing the variable-system relationships within a white-box system. Extensive experiments demonstrate that TempoGPT accurately perceives temporal information, logically infers conclusions, and achieves state-of-the-art in the constructed complex time series reasoning tasks.
  arXiv  Detail & Related papers  (2025-01-13T13:47:05Z)
• Generalized Prompt Tuning: Adapting Frozen Univariate Time Series Foundation Models for Multivariate Healthcare Time Series [3.9599054392856483]
  Time series foundation models are pre-trained on large datasets and are able to achieve state-of-the-art performance in diverse tasks. We propose a prompt-tuning-inspired fine-tuning technique, Gen-P-Tuning, that enables us to adapt an existing univariate time series foundation model. We demonstrate the effectiveness of our fine-tuning approach against various baselines on two MIMIC classification tasks, and on influenza-like illness forecasting.
  arXiv  Detail & Related papers  (2024-11-19T19:20:58Z)
• Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts [103.725112190618]
  This paper introduces Moirai-MoE, using a single input/output projection layer while delegating the modeling of diverse time series patterns to the sparse mixture of experts. Extensive experiments on 39 datasets demonstrate the superiority of Moirai-MoE over existing foundation models in both in-distribution and zero-shot scenarios.
  arXiv  Detail & Related papers  (2024-10-14T13:01:11Z)
• Towards Generalisable Time Series Understanding Across Domains [10.350643783811174]
  We introduce a novel pre-training paradigm specifically designed to handle time series heterogeneity.<n>We propose a tokeniser with learnable domain signatures, a dual masking strategy, and a normalised cross-correlation loss.<n>Our code and pre-trained weights are available at https://www.oetu.com/oetu/otis.
  arXiv  Detail & Related papers  (2024-10-09T17:09:30Z)
• Deep Time Series Models: A Comprehensive Survey and Benchmark [74.28364194333447]
  Time series data is of great significance in real-world scenarios. Recent years have witnessed remarkable breakthroughs in the time series community. We release Time Series Library (TSLib) as a fair benchmark of deep time series models for diverse analysis tasks.
  arXiv  Detail & Related papers  (2024-07-18T08:31:55Z)
• Advancing Time Series Classification with Multimodal Language Modeling [6.624754582682479]
  We propose InstructTime, a novel attempt to reshape time series classification as a learning-to-generate paradigm. The core idea is to formulate the classification of time series as a multimodal understanding task, in which both task-specific instructions and raw time series are treated as multimodal inputs. Extensive experiments are conducted over benchmark datasets, whose results uncover the superior performance of InstructTime.
  arXiv  Detail & Related papers  (2024-03-19T02:32:24Z)
• TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting [24.834846119163885]
  We propose a novel framework, TEMPO, that can effectively learn time series representations. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains.
  arXiv  Detail & Related papers  (2023-10-08T00:02:25Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.