Self-Supervised Contrastive Learning for Long-term Forecasting

• URL: http://arxiv.org/abs/2402.02023v2
• Date: Sun, 24 Mar 2024 04:01:18 GMT
• Title: Self-Supervised Contrastive Learning for Long-term Forecasting
• Authors: Junwoo Park, Daehoon Gwak, Jaegul Choo, Edward Choi,
• Abstract summary: Long-term forecasting presents unique challenges due to the time and memory complexity. Existing methods, which rely on sliding windows to process long sequences, struggle to effectively capture long-term variations. We introduce a novel approach that overcomes this limitation by employing contrastive learning and enhanced decomposition architecture.
• Score: 41.11757636744812
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Long-term forecasting presents unique challenges due to the time and memory complexity of handling long sequences. Existing methods, which rely on sliding windows to process long sequences, struggle to effectively capture long-term variations that are partially caught within the short window (i.e., outer-window variations). In this paper, we introduce a novel approach that overcomes this limitation by employing contrastive learning and enhanced decomposition architecture, specifically designed to focus on long-term variations. To this end, our contrastive loss incorporates global autocorrelation held in the whole time series, which facilitates the construction of positive and negative pairs in a self-supervised manner. When combined with our decomposition networks, our contrastive learning significantly improves long-term forecasting performance. Extensive experiments demonstrate that our approach outperforms 14 baseline models in multiple experiments over nine long-term benchmarks, especially in challenging scenarios that require a significantly long output for forecasting. Source code is available at https://github.com/junwoopark92/Self-Supervised-Contrastive-Forecsating.

Related papers

• Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting [36.577411683455786]
  Recent linear and transformer-based forecasters have shown superior performance in time series forecasting. They are constrained by their inherent inability to effectively address long-range dependencies in time series data. We introduce a fast and effective Spectral Attention mechanism, which preserves temporal correlations among samples.
  arXiv  Detail & Related papers  (2024-10-28T06:17:20Z)
• Selecting Influential Samples for Long Context Alignment via Homologous Models' Guidance and Contextual Awareness Measurement [62.87020831987625]
  We propose a novel framework designed to identify the influential and high-quality samples enriched with long-range dependency relations. We select the most challenging samples as the influential data to effectively frame the long-range dependencies. Experiments indicate that GATEAU effectively identifies samples enriched with long-range dependency relations and the model trained on these selected samples exhibits better instruction-following and long-context understanding capabilities.
  arXiv  Detail & Related papers  (2024-10-21T04:30:53Z)
• Deep End-to-End Survival Analysis with Temporal Consistency [49.77103348208835]
  We present a novel Survival Analysis algorithm designed to efficiently handle large-scale longitudinal data. A central idea in our method is temporal consistency, a hypothesis that past and future outcomes in the data evolve smoothly over time. Our framework uniquely incorporates temporal consistency into large datasets by providing a stable training signal.
  arXiv  Detail & Related papers  (2024-10-09T11:37:09Z)
• TimeBridge: Non-Stationarity Matters for Long-term Time Series Forecasting [49.6208017412376]
  TimeBridge is a novel framework designed to bridge the gap between non-stationarity and dependency modeling. TimeBridge consistently achieves state-of-the-art performance in both short-term and long-term forecasting.
  arXiv  Detail & Related papers  (2024-10-06T10:41:03Z)
• Multiscale Representation Enhanced Temporal Flow Fusion Model for Long-Term Workload Forecasting [19.426131129034115]
  This paper proposes a novel framework leveraging self-supervised multiscale representation learning to capture both long-term and near-term workload patterns. The long-term history is encoded through multiscale representations while the near-term observations are modeled via temporal flow fusion.
  arXiv  Detail & Related papers  (2024-07-29T04:42:18Z)
• Multi-Scale Dilated Convolution Network for Long-Term Time Series Forecasting [17.132063819650355]
  We propose Multi Scale Dilated Convolution Network (MSDCN) to capture the period and trend characteristics of long time series. We design different convolution blocks with exponentially growing dilations and varying kernel sizes to sample time series data at different scales. To validate the effectiveness of the proposed approach, we conduct experiments on eight challenging long-term time series forecasting benchmark datasets.
  arXiv  Detail & Related papers  (2024-05-09T02:11:01Z)
• Loss Shaping Constraints for Long-Term Time Series Forecasting [79.3533114027664]
  We present a Constrained Learning approach for long-term time series forecasting that respects a user-defined upper bound on the loss at each time-step. We propose a practical Primal-Dual algorithm to tackle it, and aims to demonstrate that it exhibits competitive average performance in time series benchmarks, while shaping the errors across the predicted window.
  arXiv  Detail & Related papers  (2024-02-14T18:20:44Z)
• Parsimony or Capability? Decomposition Delivers Both in Long-term Time Series Forecasting [46.63798583414426]
  Long-term time series forecasting (LTSF) represents a critical frontier in time series analysis. Our study demonstrates, through both analytical and empirical evidence, that decomposition is key to containing excessive model inflation. Remarkably, by tailoring decomposition to the intrinsic dynamics of time series data, our proposed model outperforms existing benchmarks.
  arXiv  Detail & Related papers  (2024-01-22T13:15:40Z)
• Effective Long-Context Scaling of Foundation Models [90.57254298730923]
  We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2.
  arXiv  Detail & Related papers  (2023-09-27T21:41:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.