Positional Encoding in Transformer-Based Time Series Models: A Survey

• URL: http://arxiv.org/abs/2502.12370v2
• Date: Thu, 18 Sep 2025 22:31:51 GMT
• Title: Positional Encoding in Transformer-Based Time Series Models: A Survey
• Authors: Habib Irani, Vangelis Metsis,
• Abstract summary: This survey systematically examines existing techniques for positional encoding in transformer-based time series models.<n>Data characteristics like sequence length, signal complexity, and dimensionality significantly influence method effectiveness.<n>We outline key challenges and suggest potential research directions to enhance positional encoding strategies.
• Score: 1.4524096882720263
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in transformer-based models have greatly improved time series analysis, providing robust solutions for tasks such as forecasting, anomaly detection, and classification. A crucial element of these models is positional encoding, which allows transformers to capture the intrinsic sequential nature of time series data. This survey systematically examines existing techniques for positional encoding in transformer-based time series models. We investigate a variety of methods, including fixed, learnable, relative, and hybrid approaches, and evaluate their effectiveness in different time series classification tasks. Our findings indicate that data characteristics like sequence length, signal complexity, and dimensionality significantly influence method effectiveness. Advanced positional encoding methods exhibit performance gains in terms of prediction accuracy, however, they come at the cost of increased computational complexity. Furthermore, we outline key challenges and suggest potential research directions to enhance positional encoding strategies. By delivering a comprehensive overview and quantitative benchmarking, this survey intends to assist researchers and practitioners in selecting and designing effective positional encoding methods for transformer-based time series models.

Related papers

• MFRS: A Multi-Frequency Reference Series Approach to Scalable and Accurate Time-Series Forecasting [51.94256702463408]
  Time series predictability is derived from periodic characteristics at different frequencies. We propose a novel time series forecasting method based on multi-frequency reference series correlation analysis. Experiments on major open and synthetic datasets show state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-11T11:40:14Z)
• VSFormer: Value and Shape-Aware Transformer with Prior-Enhanced Self-Attention for Multivariate Time Series Classification [47.92529531621406]
  We propose a novel method, VSFormer, that incorporates both discriminative patterns (shape) and numerical information (value)<n>In addition, we extract class-specific prior information derived from supervised information to enrich the positional encoding.<n>Extensive experiments on all 30 UEA archived datasets demonstrate the superior performance of our method compared to SOTA models.
  arXiv  Detail & Related papers  (2024-12-21T07:31:22Z)
• Metadata Matters for Time Series: Informative Forecasting with Transformers [70.38241681764738]
  We propose a Metadata-informed Time Series Transformer (MetaTST) for time series forecasting. To tackle the unstructured nature of metadata, MetaTST formalizes them into natural languages by pre-designed templates. A Transformer encoder is employed to communicate series and metadata tokens, which can extend series representations by metadata information.
  arXiv  Detail & Related papers  (2024-10-04T11:37:55Z)
• Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence [92.07601770031236]
  We investigate semantically meaningful patterns in the attention heads of an encoder-only Transformer architecture.<n>We find that fixing the attention weights not only accelerates the training process but also enhances the stability of the optimization.
  arXiv  Detail & Related papers  (2024-09-20T07:41:47Z)
• PRformer: Pyramidal Recurrent Transformer for Multivariate Time Series Forecasting [82.03373838627606]
  Self-attention mechanism in Transformer architecture requires positional embeddings to encode temporal order in time series prediction. We argue that this reliance on positional embeddings restricts the Transformer's ability to effectively represent temporal sequences. We present a model integrating PRE with a standard Transformer encoder, demonstrating state-of-the-art performance on various real-world datasets.
  arXiv  Detail & Related papers  (2024-08-20T01:56:07Z)
• Improving Transformers using Faithful Positional Encoding [55.30212768657544]
  We propose a new positional encoding method for a neural network architecture called the Transformer. Unlike the standard sinusoidal positional encoding, our approach has a guarantee of not losing information about the positional order of the input sequence.
  arXiv  Detail & Related papers  (2024-05-15T03:17:30Z)
• FFAD: A Novel Metric for Assessing Generated Time Series Data Utilizing Fourier Transform and Auto-encoder [9.103662085683304]
  The Fr'echet Inception Distance (FID) serves as the standard metric for evaluating generative models in image synthesis. This work proposes a novel solution leveraging the Fourier transform and Auto-encoder, termed the Fr'echet Fourier-transform Auto-encoder Distance (FFAD) Through our experimental results, we showcase the potential of FFAD for effectively distinguishing samples from different classes.
  arXiv  Detail & Related papers  (2024-03-11T10:26:04Z)
• Improving Position Encoding of Transformers for Multivariate Time Series Classification [5.467400475482668]
  We propose a new absolute position encoding method dedicated to time series data called time Absolute Position. We then propose a novel time series classification (MTSC) model combining tAPE/eRPE and convolution-based input encoding named ConvTran to improve the position and data embedding of time series data.
  arXiv  Detail & Related papers  (2023-05-26T05:30:04Z)
• Stecformer: Spatio-temporal Encoding Cascaded Transformer for Multivariate Long-term Time Series Forecasting [11.021398675773055]
  We propose a complete solution to address problems in terms of feature extraction and target prediction. For extraction, we design an efficient-temporal encoding extractor including a semi-adaptive graph to acquire sufficient-temporal information. For prediction, we propose a Cascaded De Predictor (CDP) to strengthen the correlation between different intervals.
  arXiv  Detail & Related papers  (2023-05-25T13:00:46Z)
• Ti-MAE: Self-Supervised Masked Time Series Autoencoders [16.98069693152999]
  We propose a novel framework named Ti-MAE, in which the input time series are assumed to follow an integrate distribution. Ti-MAE randomly masks out embedded time series data and learns an autoencoder to reconstruct them at the point-level. Experiments on several public real-world datasets demonstrate that our framework of masked autoencoding could learn strong representations directly from the raw data.
  arXiv  Detail & Related papers  (2023-01-21T03:20:23Z)
• A Robust and Explainable Data-Driven Anomaly Detection Approach For Power Electronics [56.86150790999639]
  We present two anomaly detection and classification approaches, namely the Matrix Profile algorithm and anomaly transformer. The Matrix Profile algorithm is shown to be well suited as a generalizable approach for detecting real-time anomalies in streaming time-series data. A series of custom filters is created and added to the detector to tune its sensitivity, recall, and detection accuracy.
  arXiv  Detail & Related papers  (2022-09-23T06:09:35Z)
• Transformers in Time Series: A Survey [66.50847574634726]
  We systematically review Transformer schemes for time series modeling by highlighting their strengths as well as limitations. From the perspective of network structure, we summarize the adaptations and modifications that have been made to Transformers. From the perspective of applications, we categorize time series Transformers based on common tasks including forecasting, anomaly detection, and classification.
  arXiv  Detail & Related papers  (2022-02-15T01:43:27Z)
• Transformers predicting the future. Applying attention in next-frame and time series forecasting [0.0]
  Recurrent Neural Networks were, until recently, one of the best ways to capture the timely dependencies in sequences. With the introduction of the Transformer, it has been proven that an architecture with only attention-mechanisms without any RNN can improve on the results in various sequence processing tasks.
  arXiv  Detail & Related papers  (2021-08-18T16:17:29Z)
• Change Point Detection in Time Series Data using Autoencoders with a Time-Invariant Representation [69.34035527763916]
  Change point detection (CPD) aims to locate abrupt property changes in time series data. Recent CPD methods demonstrated the potential of using deep learning techniques, but often lack the ability to identify more subtle changes in the autocorrelation statistics of the signal. We employ an autoencoder-based methodology with a novel loss function, through which the used autoencoders learn a partially time-invariant representation that is tailored for CPD.
  arXiv  Detail & Related papers  (2020-08-21T15:03:21Z)
• Temporal signals to images: Monitoring the condition of industrial assets with deep learning image processing algorithms [3.9023554886892438]
  This paper reviews the signal to image encoding approaches found in the literature. We propose modifications to some of their original formulations to make them more robust to the variability in large datasets. The selected encoding methods are Gramian Angular Field, Markov Transition Field, recurrence plot, grey scale encoding, spectrogram, and scalogram.
  arXiv  Detail & Related papers  (2020-05-14T14:42:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.