HoTPP Benchmark: Are We Good at the Long Horizon Events Forecasting?

• URL: http://arxiv.org/abs/2406.14341v2
• Date: Wed, 02 Oct 2024 13:24:42 GMT
• Title: HoTPP Benchmark: Are We Good at the Long Horizon Events Forecasting?
• Authors: Ivan Karpukhin, Foma Shipilov, Andrey Savchenko,
• Abstract summary: Accurately forecasting multiple future events within a given time horizon is crucial for finance, retail, social networks, and healthcare applications. We propose a novel evaluation method inspired by object detection techniques from computer vision. To support further research, we release HoTPP, the first benchmark designed explicitly for evaluating long-horizon MTPP predictions.
• Score: 1.3654846342364308
• License:
• Abstract: Accurately forecasting multiple future events within a given time horizon is crucial for finance, retail, social networks, and healthcare applications. Event timing and labels are typically modeled using Marked Temporal Point Processes (MTPP), with evaluations often focused on next-event prediction quality. While some studies have extended evaluations to a fixed number of future events, we demonstrate that this approach leads to inaccuracies in handling false positives and false negatives. To address these issues, we propose a novel evaluation method inspired by object detection techniques from computer vision. Specifically, we introduce Temporal mean Average Precision (T-mAP), a temporal variant of mAP, which overcomes the limitations of existing long-horizon evaluation metrics. Our extensive experiments demonstrate that models with strong next-event prediction accuracy can yield poor long-horizon forecasts and vice versa, indicating that specialized methods are needed for each task. To support further research, we release HoTPP, the first benchmark designed explicitly for evaluating long-horizon MTPP predictions. HoTPP includes large-scale datasets with up to 43 million events and provides optimized procedures for both autoregressive and parallel inference, paving the way for future advancements in the field.

Related papers

• DeTPP: Leveraging Object Detection for Robust Long-Horizon Event Prediction [1.534667887016089]
  We introduce DeTPP, a novel approach inspired by object detection techniques from computer vision. DeTPP employs a unique matching-based loss function that selectively prioritizes reliably predictable events. The proposed hybrid approach enhances the accuracy of next event prediction by up to 2.7% on a large transactional dataset.
  arXiv  Detail & Related papers  (2024-08-23T14:57:46Z)
• F-FOMAML: GNN-Enhanced Meta-Learning for Peak Period Demand Forecasting with Proxy Data [65.6499834212641]
  We formulate the demand prediction as a meta-learning problem and develop the Feature-based First-Order Model-Agnostic Meta-Learning (F-FOMAML) algorithm. By considering domain similarities through task-specific metadata, our model improved generalization, where the excess risk decreases as the number of training tasks increases. Compared to existing state-of-the-art models, our method demonstrates a notable improvement in demand prediction accuracy, reducing the Mean Absolute Error by 26.24% on an internal vending machine dataset and by 1.04% on the publicly accessible JD.com dataset.
  arXiv  Detail & Related papers  (2024-06-23T21:28:50Z)
• STEMO: Early Spatio-temporal Forecasting with Multi-Objective Reinforcement Learning [11.324029387605888]
  We propose an early-temporal forecasting model based on Multi-Objective reinforcement learning. Our method demonstrates superior performance on three large-scale real-world datasets.
  arXiv  Detail & Related papers  (2024-06-06T13:03:51Z)
• Enhancing Mean-Reverting Time Series Prediction with Gaussian Processes: Functional and Augmented Data Structures in Financial Forecasting [0.0]
  We explore the application of Gaussian Processes (GPs) for predicting mean-reverting time series with an underlying structure. GPs offer the potential to forecast not just the average prediction but the entire probability distribution over a future trajectory. This is particularly beneficial in financial contexts, where accurate predictions alone may not suffice if incorrect volatility assessments lead to capital losses.
  arXiv  Detail & Related papers  (2024-02-23T06:09:45Z)
• Loss Shaping Constraints for Long-Term Time Series Forecasting [79.3533114027664]
  We present a Constrained Learning approach for long-term time series forecasting that respects a user-defined upper bound on the loss at each time-step. We propose a practical Primal-Dual algorithm to tackle it, and aims to demonstrate that it exhibits competitive average performance in time series benchmarks, while shaping the errors across the predicted window.
  arXiv  Detail & Related papers  (2024-02-14T18:20:44Z)
• SMURF-THP: Score Matching-based UnceRtainty quantiFication for Transformer Hawkes Process [76.98721879039559]
  We propose SMURF-THP, a score-based method for learning Transformer Hawkes process and quantifying prediction uncertainty. Specifically, SMURF-THP learns the score function of events' arrival time based on a score-matching objective. We conduct extensive experiments in both event type prediction and uncertainty quantification of arrival time.
  arXiv  Detail & Related papers  (2023-10-25T03:33:45Z)
• Score Matching-based Pseudolikelihood Estimation of Neural Marked Spatio-Temporal Point Process with Uncertainty Quantification [59.81904428056924]
  We introduce SMASH: a Score MAtching estimator for learning markedPs with uncertainty quantification. Specifically, our framework adopts a normalization-free objective by estimating the pseudolikelihood of markedPs through score-matching. The superior performance of our proposed framework is demonstrated through extensive experiments in both event prediction and uncertainty quantification.
  arXiv  Detail & Related papers  (2023-10-25T02:37:51Z)
• Performative Time-Series Forecasting [71.18553214204978]
  We formalize performative time-series forecasting (PeTS) from a machine-learning perspective. We propose a novel approach, Feature Performative-Shifting (FPS), which leverages the concept of delayed response to anticipate distribution shifts. We conduct comprehensive experiments using multiple time-series models on COVID-19 and traffic forecasting tasks.
  arXiv  Detail & Related papers  (2023-10-09T18:34:29Z)
• Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive? [93.10694819127608]
  We propose a unified evaluation pipeline for forecasting methods with real-world perception inputs. Our in-depth study uncovers a substantial performance gap when transitioning from curated to perception-based data.
  arXiv  Detail & Related papers  (2023-06-15T17:03:14Z)
• Low Rank Forecasting [0.0]
  We consider the problem of forecasting multiple values of the future of a vector time series, using some past values. Our focus is on low rank forecasters, which break forecasting up into two steps. We introduce the concept of forecast consistency, which means that the estimates of the same value made at different times are consistent.
  arXiv  Detail & Related papers  (2021-01-29T05:59:19Z)
• Long Horizon Forecasting With Temporal Point Processes [22.896152113905856]
  Marked temporal point processes (MTPPs) have emerged as a powerful modeling machinery to characterize asynchronous events. In this paper, we develop DualTPP which is specifically well-suited to long horizon event forecasting. Experiments with a diverse set of real datasets show that DualTPP outperforms existing MTPP methods on long horizon forecasting by substantial margins.
  arXiv  Detail & Related papers  (2021-01-08T01:09:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.