Multi-Task Time Series Forecasting With Shared Attention

• URL: http://arxiv.org/abs/2101.09645v1
• Date: Sun, 24 Jan 2021 04:25:08 GMT
• Title: Multi-Task Time Series Forecasting With Shared Attention
• Authors: Zekai Chen, Jiaze E, Xiao Zhang, Hao Sheng, Xiuzheng Cheng
• Abstract summary: We propose two self-attention based sharing schemes for multi-task time series forecasting. Our proposed architectures can not only outperform the state-of-the-art single-task forecasting baselines but also outperform the RNN-based multi-task forecasting method.
• Score: 15.294939035413217
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Time series forecasting is a key component in many industrial and business decision processes and recurrent neural network (RNN) based models have achieved impressive progress on various time series forecasting tasks. However, most of the existing methods focus on single-task forecasting problems by learning separately based on limited supervised objectives, which often suffer from insufficient training instances. As the Transformer architecture and other attention-based models have demonstrated its great capability of capturing long term dependency, we propose two self-attention based sharing schemes for multi-task time series forecasting which can train jointly across multiple tasks. We augment a sequence of paralleled Transformer encoders with an external public multi-head attention function, which is updated by all data of all tasks. Experiments on a number of real-world multi-task time series forecasting tasks show that our proposed architectures can not only outperform the state-of-the-art single-task forecasting baselines but also outperform the RNN-based multi-task forecasting method.

Related papers

• UNITS: A Unified Multi-Task Time Series Model [31.675845788410246]
  We introduce UniTS, a multi-task time series model that uses task tokenization to express predictive and generative tasks within a single model. Across 38 datasets spanning human activity sensors, healthcare, engineering, and finance domains, UniTS model performs favorably against 12 forecasting models, 20 classification models, 18 anomaly detection models, and 16 imputation models.
  arXiv  Detail & Related papers  (2024-02-29T21:25:58Z)
• Unified Training of Universal Time Series Forecasting Transformers [104.56318980466742]
  We present a Masked-based Universal Time Series Forecasting Transformer (Moirai) Moirai is trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains. Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models.
  arXiv  Detail & Related papers  (2024-02-04T20:00:45Z)
• HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling for Long-Term Forecasting [18.59792043113792]
  HiMTM is a hierarchical multi-scale masked time series modeling method designed for long-term forecasting. It comprises four integral components: (1) hierarchical multi-scale transformer (HMT) to capture temporal information at different scales; (2) decoupled encoder-decoder (DED) forces the encoder to focus on feature extraction; while the decoder to focus on pretext tasks. We conduct extensive experiments on 7 mainstream datasets to prove that HiMTM has obvious advantages over contemporary self-supervised and end-to-end learning methods.
  arXiv  Detail & Related papers  (2024-01-10T09:00:03Z)
• Successive Model-Agnostic Meta-Learning for Few-Shot Fault Time Series Prognosis [3.5573601621032944]
  We introduce a novel 'pseudo meta-task' partitioning scheme that treats a continuous time period of a time series as a meta-task. Employing continuous time series as pseudo meta-tasks allows our method to extract more comprehensive features and relationships from the data. We introduce a differential algorithm to enhance the robustness of our method across different datasets.
  arXiv  Detail & Related papers  (2023-11-04T02:07:47Z)
• TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Series [57.4208255711412]
  Building on copula theory, we propose a simplified objective for the recently-introduced transformer-based attentional copulas (TACTiS) We show that the resulting model has significantly better training dynamics and achieves state-of-the-art performance across diverse real-world forecasting tasks.
  arXiv  Detail & Related papers  (2023-10-02T16:45:19Z)
• Feature Programming for Multivariate Time Series Prediction [7.0220697993232]
  We introduce the concept of programmable feature engineering for time series modeling. We propose a feature programming framework that generates large amounts of predictive features for noisy time series.
  arXiv  Detail & Related papers  (2023-06-09T20:46:55Z)
• An End-to-End Time Series Model for Simultaneous Imputation and Forecast [14.756607742477252]
  We develop an end-to-end time series model that aims to learn the inference relation and make a multiple-step ahead forecast. Our framework trains jointly two neural networks, one to learn the feature-wise correlations and the other for the modeling of temporal behaviors.
  arXiv  Detail & Related papers  (2023-06-01T15:08:22Z)
• MulT: An End-to-End Multitask Learning Transformer [66.52419626048115]
  We propose an end-to-end Multitask Learning Transformer framework, named MulT, to simultaneously learn multiple high-level vision tasks. Our framework encodes the input image into a shared representation and makes predictions for each vision task using task-specific transformer-based decoder heads.
  arXiv  Detail & Related papers  (2022-05-17T13:03:18Z)
• Multi-Task Learning with Sequence-Conditioned Transporter Networks [67.57293592529517]
  We aim to solve multi-task learning through the lens of sequence-conditioning and weighted sampling. We propose a new suite of benchmark aimed at compositional tasks, MultiRavens, which allows defining custom task combinations. Second, we propose a vision-based end-to-end system architecture, Sequence-Conditioned Transporter Networks, which augments Goal-Conditioned Transporter Networks with sequence-conditioning and weighted sampling.
  arXiv  Detail & Related papers  (2021-09-15T21:19:11Z)
• Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference [75.95287293847697]
  Two common challenges in developing multi-task models are often overlooked in literature. First, enabling the model to be inherently incremental, continuously incorporating information from new tasks without forgetting the previously learned ones (incremental learning) Second, eliminating adverse interactions amongst tasks, which has been shown to significantly degrade the single-task performance in a multi-task setup (task interference)
  arXiv  Detail & Related papers  (2020-07-24T14:44:46Z)
• Temporally Correlated Task Scheduling for Sequence Learning [143.70523777803723]
  In many applications, a sequence learning task is usually associated with multiple temporally correlated auxiliary tasks. We introduce a learnable scheduler to sequence learning, which can adaptively select auxiliary tasks for training. Our method significantly improves the performance of simultaneous machine translation and stock trend forecasting.
  arXiv  Detail & Related papers  (2020-07-10T10:28:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.