A Survey of Deep Learning and Foundation Models for Time Series Forecasting

• URL: http://arxiv.org/abs/2401.13912v1
• Date: Thu, 25 Jan 2024 03:14:07 GMT
• Title: A Survey of Deep Learning and Foundation Models for Time Series Forecasting
• Authors: John A. Miller, Mohammed Aldosari, Farah Saeed, Nasid Habib Barna, Subas Rana, I. Budak Arpinar, and Ninghao Liu
• Abstract summary: Deep learning has been successfully applied to many application domains, yet its advantages have been slow to emerge for time series forecasting. Foundation models with extensive pre-training allow models to understand patterns and acquire knowledge that can be applied to new related problems. There is ongoing research examining how to utilize or inject such knowledge into deep learning models.
• Score: 16.814826712022324
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Learning has been successfully applied to many application domains, yet its advantages have been slow to emerge for time series forecasting. For example, in the well-known Makridakis (M) Competitions, hybrids of traditional statistical or machine learning techniques have only recently become the top performers. With the recent architectural advances in deep learning being applied to time series forecasting (e.g., encoder-decoders with attention, transformers, and graph neural networks), deep learning has begun to show significant advantages. Still, in the area of pandemic prediction, there remain challenges for deep learning models: the time series is not long enough for effective training, unawareness of accumulated scientific knowledge, and interpretability of the model. To this end, the development of foundation models (large deep learning models with extensive pre-training) allows models to understand patterns and acquire knowledge that can be applied to new related problems before extensive training data becomes available. Furthermore, there is a vast amount of knowledge available that deep learning models can tap into, including Knowledge Graphs and Large Language Models fine-tuned with scientific domain knowledge. There is ongoing research examining how to utilize or inject such knowledge into deep learning models. In this survey, several state-of-the-art modeling techniques are reviewed, and suggestions for further work are provided.

Related papers

• Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations [0.0]
  This study proposes LLIAM, the Llama Lora-Integrated Autorregresive Model. Low-Rank Adaptations are used to enhance the knowledge of the model with diverse time series datasets, known as the fine-tuning phase.
  arXiv  Detail & Related papers  (2024-10-15T12:14:01Z)
• Gradual Learning: Optimizing Fine-Tuning with Partially Mastered Knowledge in Large Language Models [51.20499954955646]
  Large language models (LLMs) acquire vast amounts of knowledge from extensive text corpora during the pretraining phase. In later stages such as fine-tuning and inference, the model may encounter knowledge not covered in the initial training. We propose a two-stage fine-tuning strategy to improve the model's overall test accuracy and knowledge retention.
  arXiv  Detail & Related papers  (2024-10-08T08:35:16Z)
• PILOT: A Pre-Trained Model-Based Continual Learning Toolbox [71.63186089279218]
  This paper introduces a pre-trained model-based continual learning toolbox known as PILOT. On the one hand, PILOT implements some state-of-the-art class-incremental learning algorithms based on pre-trained models, such as L2P, DualPrompt, and CODA-Prompt. On the other hand, PILOT fits typical class-incremental learning algorithms within the context of pre-trained models to evaluate their effectiveness.
  arXiv  Detail & Related papers  (2023-09-13T17:55:11Z)
• Deep learning models for price forecasting of financial time series: A review of recent advancements: 2020-2022 [6.05458608266581]
  Deep learning models are replacing traditional statistical and machine learning models for price forecasting tasks. This review delves deeply into deep learning-based forecasting models, presenting information on model architectures, practical applications, and their respective advantages and disadvantages. The present contribution also includes potential directions for future research, such as examining the effectiveness of deep learning models with complex structures for price forecasting.
  arXiv  Detail & Related papers  (2023-04-21T03:46:09Z)
• Stop overkilling simple tasks with black-box models and use transparent models instead [57.42190785269343]
  Deep learning approaches are able to extract features autonomously from raw data. This allows for bypassing the feature engineering process. Deep learning strategies often outperform traditional models in terms of accuracy.
  arXiv  Detail & Related papers  (2023-02-06T14:28:49Z)
• Deep networks for system identification: a Survey [56.34005280792013]
  System identification learns mathematical descriptions of dynamic systems from input-output data. Main aim of the identified model is to predict new data from previous observations. We discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks.
  arXiv  Detail & Related papers  (2023-01-30T12:38:31Z)
• From Actions to Events: A Transfer Learning Approach Using Improved Deep Belief Networks [1.0554048699217669]
  This paper proposes a novel approach to map the knowledge from action recognition to event recognition using an energy-based model. Such a model can process all frames simultaneously, carrying spatial and temporal information through the learning process.
  arXiv  Detail & Related papers  (2022-11-30T14:47:10Z)
• Leveraging the structure of dynamical systems for data-driven modeling [111.45324708884813]
  We consider the impact of the training set and its structure on the quality of the long-term prediction. We show how an informed design of the training set, based on invariants of the system and the structure of the underlying attractor, significantly improves the resulting models.
  arXiv  Detail & Related papers  (2021-12-15T20:09:20Z)
• Deep Learning for Road Traffic Forecasting: Does it Make a Difference? [6.220008946076208]
  This paper focuses on critically analyzing the state of the art in what refers to the use of Deep Learning for this particular ITS research area. A posterior critical analysis is held to formulate questions and trigger a necessary debate about the issues of Deep Learning for traffic forecasting.
  arXiv  Detail & Related papers  (2020-12-02T15:56:11Z)
• Knowledge Distillation: A Survey [87.51063304509067]
  Deep neural networks have been successful in both industry and academia, especially for computer vision tasks. It is a challenge to deploy these cumbersome deep models on devices with limited resources. Knowledge distillation effectively learns a small student model from a large teacher model.
  arXiv  Detail & Related papers  (2020-06-09T21:47:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.