Fine-Tuning a Time Series Foundation Model with Wasserstein Loss

• URL: http://arxiv.org/abs/2409.15367v2
• Date: Mon, 18 Nov 2024 17:00:32 GMT
• Title: Fine-Tuning a Time Series Foundation Model with Wasserstein Loss
• Abstract summary: We show that replacing cross-entropy loss with Wasserstein loss significantly improves point estimation. Our results demonstrate that replacing cross-entropy loss with Wasserstein loss significantly improves point estimation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Inspired by recent advancements in large language models (LLMs) for Natural Language Processing (NLP), there has been a surge in research focused on developing foundational models for time series forecasting. One approach involves training LLM architectures on tokenized time series data using cross-entropy loss. Although this method has demonstrated promising results, cross-entropy loss is primarily designed for classification tasks and does not account for the distance between classes. To address this limitation, we propose using the Wasserstein loss for such architectures. To validate our approach, we fine-tuned a foundational time series model on $22$ zero-shot datasets, comparing the performance of cross-entropy loss with that of Wasserstein loss. Our results demonstrate that replacing cross-entropy loss with Wasserstein loss significantly improves point estimation.

Related papers

• Revisiting the Seasonal Trend Decomposition for Enhanced Time Series Forecasting [12.606412750084813]
  Building upon the decomposition of the time series, we enhance the architecture of machine learning models for better time series forecasting.<n>We take a different approach with the seasonal component by directly applying backbone models without any normalization or scaling procedures.<n>Our approach consistently yields positive results with around 10% MSE average reduction across four state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-02-06T23:32:47Z)
• Patch-wise Structural Loss for Time Series Forecasting [25.291749176117662]
  Patch-wise Structural (PS) loss is designed to enhance structural alignment by comparing time series at the patch level. PS loss captures nuanced structural discrepancies overlooked by traditional point-wise losses. It integrates seamlessly with point-wise loss, simultaneously addressing local structural inconsistencies and individual time-step errors.
  arXiv  Detail & Related papers  (2025-03-02T12:36:15Z)
• WinTSR: A Windowed Temporal Saliency Rescaling Method for Interpreting Time Series Deep Learning Models [0.51795041186793]
  We introduce a novel interpretation method, textitWindowed Temporal Saliency Rescaling (WinTSR) We benchmark WinTSR against 10 recent interpretation techniques with 5 state-of-the-art deep-learning models of different architectures. Our comprehensive analysis shows that WinTSR significantly outperforms other local interpretation methods in overall performance.
  arXiv  Detail & Related papers  (2024-12-05T17:15:07Z)
• A Hybrid Loss Framework for Decomposition-based Time Series Forecasting Methods: Balancing Global and Component Errors [3.92625489118339]
  We conduct a study on the impact of overall loss on existing time series methods with sequence decomposition. Our findings reveal that overall loss may introduce bias in model learning, hindering the learning of the prioritization of more significant sub-series. We propose a hybrid loss framework combining the global and component losses.
  arXiv  Detail & Related papers  (2024-11-18T07:15:23Z)
• Hierarchical Classification Auxiliary Network for Time Series Forecasting [26.92086695600799]
  We introduce a novel approach by tokenizing time series values to train forecasting models via cross-entropy loss. We propose Hierarchical Classification Auxiliary Network, HCAN, a general model-gnostic component that can be integrated with any forecasting model. Experiments integrating HCAN with state-of-the-art forecasting models demonstrate substantial improvements over baselines on several real-world datasets.
  arXiv  Detail & Related papers  (2024-05-29T10:38:25Z)
• Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting [54.04430089029033]
  We present Lag-Llama, a general-purpose foundation model for time series forecasting based on a decoder-only transformer architecture. Lag-Llama is pretrained on a large corpus of diverse time series data from several domains, and demonstrates strong zero-shot generalization capabilities. When fine-tuned on relatively small fractions of such previously unseen datasets, Lag-Llama achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-10-12T12:29:32Z)
• Multi-timestep models for Model-based Reinforcement Learning [10.940666275830052]
  In model-based reinforcement learning (MBRL), most algorithms rely on simulating trajectories from one-step dynamics models learned on data. We tackle this issue by using a multi-timestep objective to train one-step models. We find that exponentially decaying weights lead to models that significantly improve the long-horizon R2 score.
  arXiv  Detail & Related papers  (2023-10-09T12:42:39Z)
• Post-Processing Temporal Action Detection [134.26292288193298]
  Temporal Action Detection (TAD) methods typically take a pre-processing step in converting an input varying-length video into a fixed-length snippet representation sequence. This pre-processing step would temporally downsample the video, reducing the inference resolution and hampering the detection performance in the original temporal resolution. We introduce a novel model-agnostic post-processing method without model redesign and retraining.
  arXiv  Detail & Related papers  (2022-11-27T19:50:37Z)
• Deep Equilibrium Optical Flow Estimation [80.80992684796566]
  Recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms. These RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. We propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer.
  arXiv  Detail & Related papers  (2022-04-18T17:53:44Z)
• Time-Series Imputation with Wasserstein Interpolation for Optimal Look-Ahead-Bias and Variance Tradeoff [66.59869239999459]
  In finance, imputation of missing returns may be applied prior to training a portfolio optimization model. There is an inherent trade-off between the look-ahead-bias of using the full data set for imputation and the larger variance in the imputation from using only the training data. We propose a Bayesian posterior consensus distribution which optimally controls the variance and look-ahead-bias trade-off in the imputation.
  arXiv  Detail & Related papers  (2021-02-25T09:05:35Z)
• Adjusting for Autocorrelated Errors in Neural Networks for Time Series Regression and Forecasting [10.659189276058948]
  We learn the autocorrelation coefficient jointly with the model parameters in order to adjust for autocorrelated errors. For time series regression, large-scale experiments indicate that our method outperforms the Prais-Winsten method. Results across a wide range of real-world datasets show that our method enhances performance in almost all cases.
  arXiv  Detail & Related papers  (2021-01-28T04:25:51Z)
• Time Series Data Imputation: A Survey on Deep Learning Approaches [4.4458738910060775]
  Time series data imputation is a well-studied problem with different categories of methods. Time series methods based on deep learning have made progress with the usage of models like RNN. We will review and discuss their model architectures, their pros and cons as well as their effects to show the development of the time series imputation methods.
  arXiv  Detail & Related papers  (2020-11-23T11:57:27Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• Nested-Wasserstein Self-Imitation Learning for Sequence Generation [158.19606942252284]
  We propose the concept of nested-Wasserstein distance for distributional semantic matching. A novel nested-Wasserstein self-imitation learning framework is developed, encouraging the model to exploit historical high-rewarded sequences.
  arXiv  Detail & Related papers  (2020-01-20T02:19:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.