Transformer Hawkes Process

• URL: http://arxiv.org/abs/2002.09291v5
• Date: Sun, 21 Feb 2021 01:59:26 GMT
• Title: Transformer Hawkes Process
• Authors: Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, Hongyuan Zha
• Abstract summary: We propose a Transformer Hawkes Process (THP) model, which leverages the self-attention mechanism to capture long-term dependencies. THP outperforms existing models in terms of both likelihood and event prediction accuracy by a notable margin. We provide a concrete example, where THP achieves improved prediction performance for learning multiple point processes when incorporating their relational information.
• Score: 79.16290557505211
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern data acquisition routinely produce massive amounts of event sequence data in various domains, such as social media, healthcare, and financial markets. These data often exhibit complicated short-term and long-term temporal dependencies. However, most of the existing recurrent neural network based point process models fail to capture such dependencies, and yield unreliable prediction performance. To address this issue, we propose a Transformer Hawkes Process (THP) model, which leverages the self-attention mechanism to capture long-term dependencies and meanwhile enjoys computational efficiency. Numerical experiments on various datasets show that THP outperforms existing models in terms of both likelihood and event prediction accuracy by a notable margin. Moreover, THP is quite general and can incorporate additional structural knowledge. We provide a concrete example, where THP achieves improved prediction performance for learning multiple point processes when incorporating their relational information.

Related papers

• UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting [98.12558945781693]
  We propose a transformer-based model UniTST containing a unified attention mechanism on the flattened patch tokens. Although our proposed model employs a simple architecture, it offers compelling performance as shown in our experiments on several datasets for time series forecasting.
  arXiv  Detail & Related papers  (2024-06-07T14:39:28Z)
• RoTHP: Rotary Position Embedding-based Transformer Hawkes Process [0.0]
  Temporal Point Processes (TPPs) are commonly used for modeling asynchronous event sequences data. We propose a new Rotary Position Embedding-based THP architecture in this paper.
  arXiv  Detail & Related papers  (2024-05-11T10:59:09Z)
• PDETime: Rethinking Long-Term Multivariate Time Series Forecasting from the perspective of partial differential equations [49.80959046861793]
  We present PDETime, a novel LMTF model inspired by the principles of Neural PDE solvers. Our experimentation across seven diversetemporal real-world LMTF datasets reveals that PDETime adapts effectively to the intrinsic nature of the data.
  arXiv  Detail & Related papers  (2024-02-25T17:39:44Z)
• Cumulative Distribution Function based General Temporal Point Processes [49.758080415846884]
  CuFun model represents a novel approach to TPPs that revolves around the Cumulative Distribution Function (CDF) Our approach addresses several critical issues inherent in traditional TPP modeling. Our contributions encompass the introduction of a pioneering CDF-based TPP model, the development of a methodology for incorporating past event information into future event prediction.
  arXiv  Detail & Related papers  (2024-02-01T07:21:30Z)
• MPR-Net:Multi-Scale Pattern Reproduction Guided Universality Time Series Interpretable Forecasting [13.790498420659636]
  Time series forecasting has received wide interest from existing research due to its broad applications inherent challenging. This paper proposes a forecasting model, MPR-Net. It first adaptively decomposes multi-scale historical series patterns using convolution operation, then constructs a pattern extension forecasting method based on the prior knowledge of pattern reproduction, and finally reconstructs future patterns into future series using deconvolution operation. By leveraging the temporal dependencies present in the time series, MPR-Net not only achieves linear time complexity, but also makes the forecasting process interpretable.
  arXiv  Detail & Related papers  (2023-07-13T13:16:01Z)
• Discovering Predictable Latent Factors for Time Series Forecasting [39.08011991308137]
  We develop a novel framework for inferring the intrinsic latent factors implied by the observable time series. We introduce three characteristics, i.e., predictability, sufficiency, and identifiability, and model these characteristics via the powerful deep latent dynamics models. Empirical results on multiple real datasets show the efficiency of our method for different kinds of time series forecasting.
  arXiv  Detail & Related papers  (2023-03-18T14:37:37Z)
• Grouped self-attention mechanism for a memory-efficient Transformer [64.0125322353281]
  Real-world tasks such as forecasting weather, electricity consumption, and stock market involve predicting data that vary over time. Time-series data are generally recorded over a long period of observation with long sequences owing to their periodic characteristics and long-range dependencies over time. We propose two novel modules, Grouped Self-Attention (GSA) and Compressed Cross-Attention (CCA) Our proposed model efficiently exhibited reduced computational complexity and performance comparable to or better than existing methods.
  arXiv  Detail & Related papers  (2022-10-02T06:58:49Z)
• TACTiS: Transformer-Attentional Copulas for Time Series [76.71406465526454]
  estimation of time-varying quantities is a fundamental component of decision making in fields such as healthcare and finance. We propose a versatile method that estimates joint distributions using an attention-based decoder. We show that our model produces state-of-the-art predictions on several real-world datasets.
  arXiv  Detail & Related papers  (2022-02-07T21:37:29Z)
• Temporal Attention Augmented Transformer Hawkes Process [4.624987488467739]
  We come up with a new kind of Transformer-based Hawkes process model, Temporal Attention Augmented Transformer Hawkes Process (TAA-THP) We modify the traditional dot-product attention structure, and introduce the temporal encoding into attention structure. We conduct numerous experiments on a wide range of synthetic and real-life datasets to validate the performance of our proposed TAA-THP model.
  arXiv  Detail & Related papers  (2021-12-29T09:45:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.