One Fits All:Power General Time Series Analysis by Pretrained LM
- URL: http://arxiv.org/abs/2302.11939v6
- Date: Sun, 15 Oct 2023 05:07:17 GMT
- Title: One Fits All:Power General Time Series Analysis by Pretrained LM
- Authors: Tian Zhou, PeiSong Niu, Xue Wang, Liang Sun, Rong Jin
- Abstract summary: We show that pre-trained models on natural language or images can lead to a comparable or state-of-the-art performance in all main time series analysis tasks.
Our results demonstrate that pre-trained models on natural language or images can lead to a comparable or state-of-the-art performance in all main time series analysis tasks.
- Score: 23.292260325891032
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Although we have witnessed great success of pre-trained models in natural
language processing (NLP) and computer vision (CV), limited progress has been
made for general time series analysis. Unlike NLP and CV where a unified model
can be used to perform different tasks, specially designed approach still
dominates in each time series analysis task such as classification, anomaly
detection, forecasting, and few-shot learning. The main challenge that blocks
the development of pre-trained model for time series analysis is the lack of a
large amount of data for training. In this work, we address this challenge by
leveraging language or CV models, pre-trained from billions of tokens, for time
series analysis. Specifically, we refrain from altering the self-attention and
feedforward layers of the residual blocks in the pre-trained language or image
model. This model, known as the Frozen Pretrained Transformer (FPT), is
evaluated through fine-tuning on all major types of tasks involving time
series. Our results demonstrate that pre-trained models on natural language or
images can lead to a comparable or state-of-the-art performance in all main
time series analysis tasks, as illustrated in Figure 1. We also found both
theoretically and empirically that the self-attention module behaviors
similarly to principle component analysis (PCA), an observation that helps
explains how transformer bridges the domain gap and a crucial step towards
understanding the universality of a pre-trained transformer.The code is
publicly available at https://github.com/DAMO-DI-ML/One_Fits_All.
Related papers
- UNITS: A Unified Multi-Task Time Series Model [31.675845788410246]
We introduce UniTS, a multi-task time series model that uses task tokenization to express predictive and generative tasks within a single model.
Across 38 datasets spanning human activity sensors, healthcare, engineering, and finance domains, UniTS model performs favorably against 12 forecasting models, 20 classification models, 18 anomaly detection models, and 16 imputation models.
arXiv Detail & Related papers (2024-02-29T21:25:58Z) - Unified Training of Universal Time Series Forecasting Transformers [104.56318980466742]
We present a Masked-based Universal Time Series Forecasting Transformer (Moirai)
Moirai is trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains.
Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models.
arXiv Detail & Related papers (2024-02-04T20:00:45Z) - Timer: Generative Pre-trained Transformers Are Large Time Series Models [83.03091523806668]
This paper aims at the early development of large time series models (LTSM)
During pre-training, we curate large-scale datasets with up to 1 billion time points.
To meet diverse application needs, we convert forecasting, imputation, and anomaly detection of time series into a unified generative task.
arXiv Detail & Related papers (2024-02-04T06:55:55Z) - One Fits All: Universal Time Series Analysis by Pretrained LM and
Specially Designed Adaptors [23.292260325891032]
We introduce four unique adapters, designed specifically for downstream tasks based on the pre-trained model.
These adapters are further enhanced with efficient parameter tuning, resulting in superior performance compared to all state-of-the-art methods.
arXiv Detail & Related papers (2023-11-24T16:32:47Z) - Lag-Llama: Towards Foundation Models for Probabilistic Time Series
Forecasting [54.04430089029033]
We present Lag-Llama, a general-purpose foundation model for time series forecasting based on a decoder-only transformer architecture.
Lag-Llama is pretrained on a large corpus of diverse time series data from several domains, and demonstrates strong zero-shot generalization capabilities.
When fine-tuned on relatively small fractions of such previously unseen datasets, Lag-Llama achieves state-of-the-art performance.
arXiv Detail & Related papers (2023-10-12T12:29:32Z) - Toward a Foundation Model for Time Series Data [34.1973242428317]
A foundation model is a machine learning model trained on a large and diverse set of data.
We develop an effective time series foundation model by leveraging unlabeled samples from multiple domains.
arXiv Detail & Related papers (2023-10-05T21:44:50Z) - Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
Time series forecasting holds significant importance in many real-world dynamic systems.
We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting.
Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
arXiv Detail & Related papers (2023-10-03T01:31:25Z) - Ti-MAE: Self-Supervised Masked Time Series Autoencoders [16.98069693152999]
We propose a novel framework named Ti-MAE, in which the input time series are assumed to follow an integrate distribution.
Ti-MAE randomly masks out embedded time series data and learns an autoencoder to reconstruct them at the point-level.
Experiments on several public real-world datasets demonstrate that our framework of masked autoencoding could learn strong representations directly from the raw data.
arXiv Detail & Related papers (2023-01-21T03:20:23Z) - A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis [90.24921443175514]
We focus on aspect-based sentiment analysis, which involves extracting aspect term, category, and predicting their corresponding polarities.
We propose to reformulate the extraction and prediction tasks into the sequence generation task, using a generative language model with unidirectional attention.
Our approach outperforms the previous state-of-the-art (based on BERT) on average performance by a large margins in few-shot and full-shot settings.
arXiv Detail & Related papers (2022-04-11T18:31:53Z) - Train No Evil: Selective Masking for Task-Guided Pre-Training [97.03615486457065]
We propose a three-stage framework by adding a task-guided pre-training stage with selective masking between general pre-training and fine-tuning.
We show that our method can achieve comparable or even better performance with less than 50% of cost.
arXiv Detail & Related papers (2020-04-21T03:14:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.