Gated Transformer Networks for Multivariate Time Series Classification
- URL: http://arxiv.org/abs/2103.14438v1
- Date: Fri, 26 Mar 2021 12:43:32 GMT
- Title: Gated Transformer Networks for Multivariate Time Series Classification
- Authors: Minghao Liu, Shengqi Ren, Siyuan Ma, Jiahui Jiao, Yizhou Chen,
Zhiguang Wang, Wei Song
- Abstract summary: Gated Transformer Networks (GTN) is a simple extension of Transformer Networks for time series classification problem.
We conduct comprehensive experiments on thirteen dataset with full ablation study.
Our results show that GTN is able to achieve competing results with current state-of-the-art deep learning models.
- Score: 11.388531556672529
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Deep learning model (primarily convolutional networks and LSTM) for time
series classification has been studied broadly by the community with the wide
applications in different domains like healthcare, finance, industrial
engineering and IoT. Meanwhile, Transformer Networks recently achieved frontier
performance on various natural language processing and computer vision tasks.
In this work, we explored a simple extension of the current Transformer
Networks with gating, named Gated Transformer Networks (GTN) for the
multivariate time series classification problem. With the gating that merges
two towers of Transformer which model the channel-wise and step-wise
correlations respectively, we show how GTN is naturally and effectively
suitable for the multivariate time series classification task. We conduct
comprehensive experiments on thirteen dataset with full ablation study. Our
results show that GTN is able to achieve competing results with current
state-of-the-art deep learning models. We also explored the attention map for
the natural interpretability of GTN on time series modeling. Our preliminary
results provide a strong baseline for the Transformer Networks on multivariate
time series classification task and grounds the foundation for future research.
Related papers
- PRformer: Pyramidal Recurrent Transformer for Multivariate Time Series Forecasting [82.03373838627606]
Self-attention mechanism in Transformer architecture requires positional embeddings to encode temporal order in time series prediction.
We argue that this reliance on positional embeddings restricts the Transformer's ability to effectively represent temporal sequences.
We present a model integrating PRE with a standard Transformer encoder, demonstrating state-of-the-art performance on various real-world datasets.
arXiv Detail & Related papers (2024-08-20T01:56:07Z) - Timer: Generative Pre-trained Transformers Are Large Time Series Models [83.03091523806668]
This paper aims at the early development of large time series models (LTSM)
During pre-training, we curate large-scale datasets with up to 1 billion time points.
To meet diverse application needs, we convert forecasting, imputation, and anomaly detection of time series into a unified generative task.
arXiv Detail & Related papers (2024-02-04T06:55:55Z) - iTransformer: Inverted Transformers Are Effective for Time Series Forecasting [62.40166958002558]
We propose iTransformer, which simply applies the attention and feed-forward network on the inverted dimensions.
The iTransformer model achieves state-of-the-art on challenging real-world datasets.
arXiv Detail & Related papers (2023-10-10T13:44:09Z) - U-shaped Transformer: Retain High Frequency Context in Time Series
Analysis [0.5710971447109949]
In this paper, we consider the low-pass characteristics of transformers and try to incorporate the advantages of them.
We introduce patch merge and split operation to extract features with different scales and use larger datasets to fully make use of the transformer backbone.
Our experiments demonstrate that the model performs at an advanced level across multiple datasets with relatively low cost.
arXiv Detail & Related papers (2023-07-18T07:15:26Z) - A Comprehensive Survey on Applications of Transformers for Deep Learning
Tasks [60.38369406877899]
Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data.
transformer models excel in handling long dependencies between input sequence elements and enable parallel processing.
Our survey encompasses the identification of the top five application domains for transformer-based models.
arXiv Detail & Related papers (2023-06-11T23:13:51Z) - FormerTime: Hierarchical Multi-Scale Representations for Multivariate
Time Series Classification [53.55504611255664]
FormerTime is a hierarchical representation model for improving the classification capacity for the multivariate time series classification task.
It exhibits three aspects of merits: (1) learning hierarchical multi-scale representations from time series data, (2) inheriting the strength of both transformers and convolutional networks, and (3) tacking the efficiency challenges incurred by the self-attention mechanism.
arXiv Detail & Related papers (2023-02-20T07:46:14Z) - Investigation of Network Architecture for Multimodal Head-and-Neck Tumor
Segmentation [9.441769048218955]
In this study, we analyze, two recently published Transformer-based network architectures for the task of multimodal head-and-tumor segmentation.
Our results showed that modeling long-range dependencies may be helpful in cases where large structures are present and/or large field of view is needed.
For small structures such as head-and-neck tumor, the convolution-based U-Net architecture seemed to perform well, especially when training dataset is small and computational resource is limited.
arXiv Detail & Related papers (2022-12-21T02:35:46Z) - W-Transformers : A Wavelet-based Transformer Framework for Univariate
Time Series Forecasting [7.075125892721573]
We build a transformer model for non-stationary time series using wavelet-based transformer encoder architecture.
We evaluate our framework on several publicly available benchmark time series datasets from various domains.
arXiv Detail & Related papers (2022-09-08T17:39:38Z) - Transformers in Time Series: A Survey [66.50847574634726]
We systematically review Transformer schemes for time series modeling by highlighting their strengths as well as limitations.
From the perspective of network structure, we summarize the adaptations and modifications that have been made to Transformers.
From the perspective of applications, we categorize time series Transformers based on common tasks including forecasting, anomaly detection, and classification.
arXiv Detail & Related papers (2022-02-15T01:43:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.