Deep Fourier Kernel for Self-Attentive Point Processes

• URL: http://arxiv.org/abs/2002.07281v5
• Date: Sun, 21 Feb 2021 05:33:59 GMT
• Title: Deep Fourier Kernel for Self-Attentive Point Processes
• Authors: Shixiang Zhu, Minghe Zhang, Ruyi Ding, Yao Xie
• Abstract summary: We present a novel attention-based model for discrete event data to capture complex non-linear temporal dependence structures. We introduce a novel score function using Fourier kernel embedding, whose spectrum is represented using neural networks. We establish our approach's theoretical properties and demonstrate our approach's competitive performance compared to the state-of-the-art for synthetic and real data.
• Score: 16.63706478353667
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a novel attention-based model for discrete event data to capture complex non-linear temporal dependence structures. We borrow the idea from the attention mechanism and incorporate it into the point processes' conditional intensity function. We further introduce a novel score function using Fourier kernel embedding, whose spectrum is represented using neural networks, which drastically differs from the traditional dot-product kernel and can capture a more complex similarity structure. We establish our approach's theoretical properties and demonstrate our approach's competitive performance compared to the state-of-the-art for synthetic and real data.

Related papers

• Machine Learning-Enhanced Characterisation of Structured Spectral Densities: Leveraging the Reaction Coordinate Mapping [41.94295877935867]
  Spectral densities encode essential information about system-environment interactions in open-quantum systems. We leverage machine learning techniques to reconstruct key environmental features using the reaction coordinate mapping. For a dissipative spin-boson model with a structured spectral density expressed as a sum of Lorentzian peaks, we demonstrate that the time evolution of a system observable can be used by a neural network to classify the spectral density as comprising one, two, or three Lorentzian peaks.
  arXiv  Detail & Related papers  (2025-01-13T17:02:04Z)
• Nonstationary Sparse Spectral Permanental Process [24.10531062895964]
  We propose a novel approach utilizing the sparse spectral representation of nonstationary kernels. This technique relaxes the constraints on kernel types and stationarity, allowing for more flexible modeling. Experimental results on both synthetic and real-world datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-10-04T16:40:56Z)
• Capturing dynamical correlations using implicit neural representations [85.66456606776552]
  We develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data.
  arXiv  Detail & Related papers  (2023-04-08T07:55:36Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Linear Self-Attention Approximation via Trainable Feedforward Kernel [77.34726150561087]
  In pursuit of faster computation, Efficient Transformers demonstrate an impressive variety of approaches. We aim to expand the idea of trainable kernel methods to approximate the self-attention mechanism of the Transformer architecture.
  arXiv  Detail & Related papers  (2022-11-08T08:14:11Z)
• Convex Analysis of the Mean Field Langevin Dynamics [49.66486092259375]
  convergence rate analysis of the mean field Langevin dynamics is presented. $p_q$ associated with the dynamics allows us to develop a convergence theory parallel to classical results in convex optimization.
  arXiv  Detail & Related papers  (2022-01-25T17:13:56Z)
• An Empirical Study: Extensive Deep Temporal Point Process [61.14164208094238]
  We first review recent research emphasis and difficulties in modeling asynchronous event sequences with deep temporal point process. We propose a Granger causality discovery framework for exploiting the relations among multi-types of events.
  arXiv  Detail & Related papers  (2021-10-19T10:15:00Z)
• Neural Spectral Marked Point Processes [18.507050473968985]
  We introduce a novel and general neural network-based non-stationary influence kernel for handling complex discrete events. We demonstrate the superior performance of our proposed method compared with the state-of-the-art on synthetic and real data.
  arXiv  Detail & Related papers  (2021-06-20T23:00:37Z)
• A Temporal Kernel Approach for Deep Learning with Continuous-time Information [18.204325860752768]
  Sequential deep learning models such as RNN, causal CNN and attention mechanism do not readily consume continuous-time information. Discretizing the temporal data, as we show, causes inconsistency even for simple continuous-time processes. We provide a principled way to characterize continuous-time systems using deep learning tools.
  arXiv  Detail & Related papers  (2021-03-28T20:13:53Z)
• UNIPoint: Universally Approximating Point Processes Intensities [125.08205865536577]
  We provide a proof that a class of learnable functions can universally approximate any valid intensity function. We implement UNIPoint, a novel neural point process model, using recurrent neural networks to parameterise sums of basis function upon each event.
  arXiv  Detail & Related papers  (2020-07-28T09:31:56Z)
• Convolutional Spectral Kernel Learning [21.595130250234646]
  We build an interpretable convolutional spectral kernel network (textttCSKN) based on the inverse Fourier transform. We derive the generalization error bounds and introduce two regularizers to improve the performance. Experiments results on real-world datasets validate the effectiveness of the learning framework.
  arXiv  Detail & Related papers  (2020-02-28T14:35:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.