STEP: Scientific Time-Series Encoder Pretraining via Cross-Domain Distillation

• URL: http://arxiv.org/abs/2603.18688v1
• Date: Thu, 19 Mar 2026 09:47:26 GMT
• Title: STEP: Scientific Time-Series Encoder Pretraining via Cross-Domain Distillation
• Abstract summary: Scientific time series are central to scientific AI but are typically sparse, highly heterogeneous, and limited in scale.<n>We propose STEP, a Scientific Time Series Pretraining framework via cross domain distillation.<n>By combining complementary representations across different domains, STEP learns general-purpose and transferable features tailored for scientific signals.
• Score: 33.980112187180985
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Scientific time series are central to scientific AI but are typically sparse, highly heterogeneous, and limited in scale, making unified representation learning particularly challenging. Meanwhile, foundation models pretrained on relevant time series domains such as audio, general time series, and brain signals contain rich knowledge, but their applicability to scientific signals remains underexplored. In this paper, we investigate the transferability and complementarity of foundation models from relevant time series domains, and study how to effectively leverage them to build a unified encoder for scientific time series. We first systematically evaluate relevant foundation models, showing the effectiveness of knowledge transfer to scientific tasks and their complementary strengths. Based on this observation, we propose STEP, a Scientific Time Series Encoder Pretraining framework via cross domain distillation. STEP introduces adaptive patching to handle extreme-length sequences and a statistics compensation scheme to accommodate diverse numerical scales. It further leverages cross-domain distillation to integrate knowledge from multiple foundation models into a unified encoder. By combining complementary representations across different domains, STEP learns general-purpose and transferable features tailored for scientific signals. Experiments on seven scientific time series tasks demonstrate that STEP provides both an effective structure and an effective pretraining paradigm, taking a STEP toward scientific time series representation learning.

Related papers

• Intern-S2-Preview: Scientific Agentic Foundation Model [199.88087357217503]
  Intern-S2-Preview is designed to support multimodal scientific understanding, reasoning, generation, and long-horizon tasks.<n>The training pipeline begins with scientific multimodal pre-training over rendered scientific documents, interleaved image-text data, and diverse scientific corpora.<n>This pipeline is supported by practical techniques that improve rollout and training stability and efficiency.
  arXiv  Detail & Related papers  (2026-08-13T17:31:28Z)
• UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification [5.071106490524274]
  We adapt DINOv2-style self-distillation to pretrain a time series foundation model.<n>We build on the Mantis tokenizer and transformer encoder architecture as our backbone.<n>Our method achieves state-of-the-art classification performance on both UCR and UEA benchmarks.
  arXiv  Detail & Related papers  (2026-03-02T01:02:09Z)
• FusAD: Time-Frequency Fusion with Adaptive Denoising for General Time Series Analysis [92.23551599659186]
  Time series analysis plays a vital role in fields such as finance, healthcare, industry, and meteorology.<n>FusAD is a unified analysis framework designed for diverse time series tasks.
  arXiv  Detail & Related papers  (2025-12-16T04:34:27Z)
• SigTime: Learning and Visually Explaining Time Series Signatures [22.200677868580204]
  We introduce a novel learning framework that jointly trains two Transformer models using complementary time series representations.<n>The learned shapelets serve as interpretable signatures that differentiate time series across classification labels.<n>We develop a visual analytics system -- SigTIme -- with coordinated views to facilitate exploration of time series signatures.
  arXiv  Detail & Related papers  (2025-12-12T22:47:34Z)
• A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers [251.23085679210206]
  Scientific Large Language Models (Sci-LLMs) are transforming how knowledge is represented, integrated, and applied in scientific research.<n>This survey reframes the development of Sci-LLMs as a co-evolution between models and their underlying data substrate.<n>We formulate a unified taxonomy of scientific data and a hierarchical model of scientific knowledge.
  arXiv  Detail & Related papers  (2025-08-28T18:30:52Z)
• Multivariate Long-term Time Series Forecasting with Fourier Neural Filter [42.60778405812048]
  We introduce FNF as the backbone and DBD as architecture to provide excellent learning capabilities and optimal learning pathways for spatial-temporal modeling.<n>We show that FNF unifies local time-domain and global frequency-domain information processing within a single backbone that extends naturally to spatial modeling.
  arXiv  Detail & Related papers  (2025-06-10T18:40:20Z)
• UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines [64.84631333071728]
  We introduce bfUnistage, a unified Transformer-based framework fortemporal modeling.<n>Our work demonstrates that a task-specific vision-text can build a generalizable model fortemporal learning.<n>We also introduce a temporal module to incorporate temporal dynamics explicitly.
  arXiv  Detail & Related papers  (2025-03-26T17:33:23Z)
• General Time-series Model for Universal Knowledge Representation of Multivariate Time-Series data [61.163542597764796]
  We show that time series with different time granularities (or corresponding frequency resolutions) exhibit distinct joint distributions in the frequency domain.<n>A novel Fourier knowledge attention mechanism is proposed to enable learning time-aware representations from both the temporal and frequency domains.<n>An autoregressive blank infilling pre-training framework is incorporated to time series analysis for the first time, leading to a generative tasks agnostic pre-training strategy.
  arXiv  Detail & Related papers  (2025-02-05T15:20:04Z)
• Unleashing The Power of Pre-Trained Language Models for Irregularly Sampled Time Series [22.87452807636833]
  We propose a unified PLM-based framework, named ISTS-PLM, to address diverse ISTS analytical tasks.<n>It integrates novel time-aware and variable-aware PLMs tailored to tackle the intractable intra- and inter-time series modeling in ISTS.
  arXiv  Detail & Related papers  (2024-08-12T14:22:14Z)
• UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting [59.11817101030137]
  This research advocates for a unified model paradigm that transcends domain boundaries. Learning an effective cross-domain model presents the following challenges. We propose UniTime for effective cross-domain time series learning.
  arXiv  Detail & Related papers  (2023-10-15T06:30:22Z)
• TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting [24.834846119163885]
  We propose a novel framework, TEMPO, that can effectively learn time series representations. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains.
  arXiv  Detail & Related papers  (2023-10-08T00:02:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.