Transformers with Attentive Federated Aggregation for Time Series Stock Forecasting

• URL: http://arxiv.org/abs/2402.06638v1
• Date: Mon, 22 Jan 2024 07:33:28 GMT
• Title: Transformers with Attentive Federated Aggregation for Time Series Stock Forecasting
• Authors: Chu Myaet Thwal, Ye Lin Tun, Kitae Kim, Seong-Bae Park, Choong Seon Hong
• Abstract summary: Time series modeling has led to the widespread use of transformers in many time series applications. The adaptation of transformers to time series forecasting has remained limited, with both promising and inconsistent results. We propose attentive federated transformers for time series stock forecasting with better performance while preserving the privacy of participating enterprises.
• Score: 15.968396756506236
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent innovations in transformers have shown their superior performance in natural language processing (NLP) and computer vision (CV). The ability to capture long-range dependencies and interactions in sequential data has also triggered a great interest in time series modeling, leading to the widespread use of transformers in many time series applications. However, being the most common and crucial application, the adaptation of transformers to time series forecasting has remained limited, with both promising and inconsistent results. In contrast to the challenges in NLP and CV, time series problems not only add the complexity of order or temporal dependence among input sequences but also consider trend, level, and seasonality information that much of this data is valuable for decision making. The conventional training scheme has shown deficiencies regarding model overfitting, data scarcity, and privacy issues when working with transformers for a forecasting task. In this work, we propose attentive federated transformers for time series stock forecasting with better performance while preserving the privacy of participating enterprises. Empirical results on various stock data from the Yahoo! Finance website indicate the superiority of our proposed scheme in dealing with the above challenges and data heterogeneity in federated learning.

Related papers

• LSEAttention is All You Need for Time Series Forecasting [0.0]
  Transformer-based architectures have achieved remarkable success in natural language processing and computer vision. I introduce textbfLSEAttention, an approach designed to address entropy collapse and training instability commonly observed in transformer models.
  arXiv  Detail & Related papers  (2024-10-31T09:09:39Z)
• Timer-XL: Long-Context Transformers for Unified Time Series Forecasting [67.83502953961505]
  We present Timer-XL, a generative Transformer for unified time series forecasting. Timer-XL achieves state-of-the-art performance across challenging forecasting benchmarks through a unified approach.
  arXiv  Detail & Related papers  (2024-10-07T07:27:39Z)
• Unified Training of Universal Time Series Forecasting Transformers [104.56318980466742]
  We present a Masked-based Universal Time Series Forecasting Transformer (Moirai) Moirai is trained on our newly introduced Large-scale Open Time Series Archive (LOTSA) featuring over 27B observations across nine domains. Moirai achieves competitive or superior performance as a zero-shot forecaster when compared to full-shot models.
  arXiv  Detail & Related papers  (2024-02-04T20:00:45Z)
• U-shaped Transformer: Retain High Frequency Context in Time Series Analysis [0.5710971447109949]
  In this paper, we consider the low-pass characteristics of transformers and try to incorporate the advantages of them. We introduce patch merge and split operation to extract features with different scales and use larger datasets to fully make use of the transformer backbone. Our experiments demonstrate that the model performs at an advanced level across multiple datasets with relatively low cost.
  arXiv  Detail & Related papers  (2023-07-18T07:15:26Z)
• Robust representations of oil wells' intervals via sparse attention mechanism [2.604557228169423]
  We introduce the class of efficient Transformers named Regularized Transformers (Reguformers) The focus in our experiments is on oil&gas data, namely, well logs. To evaluate our models for such problems, we work with an industry-scale open dataset consisting of well logs of more than 20 wells.
  arXiv  Detail & Related papers  (2022-12-29T09:56:33Z)
• W-Transformers : A Wavelet-based Transformer Framework for Univariate Time Series Forecasting [7.075125892721573]
  We build a transformer model for non-stationary time series using wavelet-based transformer encoder architecture. We evaluate our framework on several publicly available benchmark time series datasets from various domains.
  arXiv  Detail & Related papers  (2022-09-08T17:39:38Z)
• CLMFormer: Mitigating Data Redundancy to Revitalize Transformer-based Long-Term Time Series Forecasting System [46.39662315849883]
  Long-term time-series forecasting (LTSF) plays a crucial role in various practical applications. Existing Transformer-based models, such as Fedformer and Informer, often achieve their best performances on validation sets after just a few epochs. We propose a novel approach to address this issue by employing curriculum learning and introducing a memory-driven decoder.
  arXiv  Detail & Related papers  (2022-07-16T04:05:15Z)
• Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting [86.33543833145457]
  We propose Non-stationary Transformers as a generic framework with two interdependent modules: Series Stationarization and De-stationary Attention. Our framework consistently boosts mainstream Transformers by a large margin, which reduces MSE by 49.43% on Transformer, 47.34% on Informer, and 46.89% on Reformer.
  arXiv  Detail & Related papers  (2022-05-28T12:27:27Z)
• Transformers in Time Series: A Survey [66.50847574634726]
  We systematically review Transformer schemes for time series modeling by highlighting their strengths as well as limitations. From the perspective of network structure, we summarize the adaptations and modifications that have been made to Transformers. From the perspective of applications, we categorize time series Transformers based on common tasks including forecasting, anomaly detection, and classification.
  arXiv  Detail & Related papers  (2022-02-15T01:43:27Z)
• Transformer Hawkes Process [79.16290557505211]
  We propose a Transformer Hawkes Process (THP) model, which leverages the self-attention mechanism to capture long-term dependencies. THP outperforms existing models in terms of both likelihood and event prediction accuracy by a notable margin. We provide a concrete example, where THP achieves improved prediction performance for learning multiple point processes when incorporating their relational information.
  arXiv  Detail & Related papers  (2020-02-21T13:48:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.